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cross-platform database 
apid SOL? and Embarcadero? Change 
del ver results in less than 60 minutes. 
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Embarcadero products let you access your databases from a single 
application window — so you can manage multiple databases, across 
different DBMS platforms. No other products or native tools let you do 
that. So, why waste your time on anything else? 


Try it for FREE and see for yourself! We are so sure that you can 
achieve real benefits in real time that we're offering you the chance to 
try all of Embarcadero's products for FREE in a 14 day trial. Just visit 
www.embarcadero.com/challenge to download your copy now! 
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1e hour to rock your world 


Embarcadero Technologies is throwing down 

the gauntlet to all database administrators, 

developers and data modelers. We have 

compiled a number of common database 

problems and each task has an estimated time for completion 
— but we challenge you to do them even faster! 


To get started, simply download your evaluation software and 
click on the ‘db FEST - Take the Challenge’ icon to check out the 
step-by-step guide. Then simply record the time taken to complete 
each task, return the form to us and you will automatically receive 
a FREE Embarcadero ‘db FEST tour’ t-shirt — and the chance to 
win an even bigger prize! 
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Calculate Percentiles 


—lItzik Ben-Gan 
Use T-SQL to implement statistical calculations 
such as percentile and percentile rank. 


Integrate MySQL 
and SQL Server 
—Robert Sheldon 


Need to retrieve MySQL data and insert it into a SQL 
Server database? SQL Server Integration Services and 


MySQL connectors let you do so fairly easily. 


Finding Your Top 10 
SQL Server Queries 
—Andrew J. Kelly 


Want to parse server-side trace information so that 


you can aggregate the data to find your worst- 


performing and most-called queries? The user-defined 


SQL_Signature function should do the trick. 


a 
e^ 


12 Ola Hallengren, Christian Mercure, Gregory A. Larsen, and 
Shaunt Khaldtiance share their real-world solutions for maintaining 
databases, modifying words in tables, finding the worst-performing 
T-SQL statements, and determining how much space is being used 
by tables and their indexes. 


i LJ 


Efficient Data Binding with 
Windows Presentation 
Foundation and Visual 

Studio 2008 

—Ken Spencer 

A free tool helps you use Visual Studio 2008 to automate 
wiring your SQL Server data source with a Windows 
Presentation Foundation (WPF) front end. 


i Editor's Note 


in. ut 
Going to SQL Server 
Connections in Las Vegas 
November 10—13? Stop by 
the SQL Server Magazine and Windows IT Pro 
booth. We'd love to chat. Follow our blogs from 


the conference floor at www.sql.mag.com/blogs. 
—Sheila Molnar, executive editor 
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SQLScripter 


—Kevin Kline 
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Use SQLScripter to automate database schema scripting tasks. 


IN EVERY ISSUE 


1 Editorial: 
Browser Wars Redux 


—Michael Otey 


8 SQLMAG.com 

Community Dialog: 

The Wisdom of SQL Mag 

Readers 

—Anne Grubb 

SQL Server Magazine readers are 
sharing their expertise and helping 
their peers in the comments section 
of online articles. 


PRODUCTS 


41 Product Review: 
LINQPad 


—Dan Hanan 

The free LINQPad tool lets you 
interactively query SQL Server 
using LINQ. 


Product Review: 

RowGen 2.0 

—William Sheldon 

Use RowGen 2.0 to create referen- 
tially valid but random sample data 
for use in your test environment. 


Product Review: 
dbWatch 8.1 


—John Green 

If you want affordable, basic 
database-monitoring capabilities, 
the dbWatch 8.1 tool will meet your 
needs. 
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MarketPlace 


The Back Page: 
SQL Server 2008 Editions 


—Michael Otey 


Your Savvy Assistant 
—Christan Humphries 
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Industry News: 

Bytes from the Blog 

The recent crash of the London 
Stock Exchange’s TradElect trading 
platform, which was developed and 
highly publicized by Microsoft, Ac- 
centure, and Avanade, leaves stock 
traders and industry observers 
wondering who—or what—is to 
blame. 


New Products 


Check out new and improved SQL 
Server—related products from 
Embarcadero Technologies, Sybase, 
Tableau Software, and Microsoft. 
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XMLSpy XML Editor 


Altova® XMLSpy® 2008 is the world’s best-selling XML development environment for modeling, editing, 


transforming, debugging, and profiling all XML technologies. Download a free trial today! 


XMLSpy provides unsurpassed compliance with the 
puppor forvarylarge XMT nies latest industry standards. It even supports the new 
Office Open XML (OOXML) formats utilized by 


Microsoft® Office 2007. The advanced functionality 


e Detailed find & replace in XML Schema editor 
e Enhanced XSLT details in Info Window 

e XSL Outline entry helper 

e Visual Studio support updated for VS 2008 


e Support for Java, C#, JavaScript, and 
VBScript in stylesheets 


in XMLSpy is coupled with user-friendly views and 
entry helpers, wizards, and debuggers, helping 


developers create, edit, and optimize today’s most 


advanced XML-based applications and Web services 
e And much more... 
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Browser Wars Redux 


oogle’s recent beta release of the Chrome 

browser has reignited the fight for control of 
the web. It’s no secret that Microsoft has owned the 
browser space ever since it vanquished Netscape 
several years ago by bundling Internet Explorer 
(IE) with the Windows OS. The emergence of the 
Mozilla Firefox browser never really threatened 
Microsoft. Although Firefox has gained about 20 
percent global market share, it’s never mounted a 
serious bid to overtake IE. However, the continued 
success of Firefox underscores the dissatisfaction 
that businesses and end users have with IE, and 
perhaps with Microsoft in general. 


Why Is Google 

in the Browser Business? 

Google is a far cry from little, unthreatening, 
open-source Mozilla. Google’s release of Chrome 
is nothing short of a shot across Microsoft’s bow. 
Microsoft sees Google as its primary competi- 
tor: Google’s web-only orientation threatens to 
undermine Microsoft’s desktop dominance by 
supplanting the use of standalone software with 
web-based tools and services. Although we're not 
quite there yet, it’s clear that cloud computing and 
Software as a Service (SaaS) are powerful emerg- 
ing trends in the tech industry. The web browser 
enables SaaS (or Software Plus Services—S+S— 
if you feel particularly Microsoft-inclined) and 
cloud computing. 

Although I didn’t expect it, I can see why 
Google wanted to develop a browser of its own. 
Google’s offerings are exclusively web-based and 
most users get to them via Microsoft’s IE. This 
leaves control of Google’s key web access technol- 
ogy in the hands of its biggest competitor. By cre- 
ating Chrome, Google attempts to wrest control of 
this critical link from Microsoft and remove its de- 
pendence on its rival’s browser. Microsoft will cer- 
tainly want to maintain its market position on the 
web and on the desktop. The software giant will 
continue to promote software-oriented solutions. 
This could deter Microsoft from whole-heartedly 
supporting future web technologies that might 
threaten its core business lines. 


SQL Server Magazine * www.sqlmag.com 


Does Chrome have a shot at the title? If the 
continued success of Firefox is any indication, I 
think it does. At first, to even know about Firefox 
you had to be a bit tech savvy—even finding it was 
a challenge. Not so with Chrome. Everyone who 
uses Google (and isn't that everyone?) will be ex- 
posed to Chrome. Chrome is sure to have a major 
impact on IE and Firefox users. 


Chrome's Impact 
on SQL Server Development 

Although SQL Server isn't at the forefront of this 
epic battle, there's no doubt that SQL Server devel- 
opers will be affected by 
the new Chrome brows- 
er. Today, the majority 
of SQL Server applica- 


Michael Otey 


(mikeo@ sqlmag.com) is technical director 
for Windows IT Pro and SQL Server Maga- 
zine and author of Microsoft SQL Server 

2008 New Features (Osborne/McGraw-Hill). 


Although SQL Server isn’t 


tions are created with 
ASP and ASP.NET. The 
emergence of Chrome, 
with its revved up V8 
JavaScript engine, has 


at the forefront of this epic 
battle, there’s no doubt 
that SQL Server developers 
will be affected by the new 


the potential to reshape 
the face of web applica- 
tions. It could affect the 
future of web development technologies that will 
need to access SQL Server’s database services on 
the backend. 


Who Wins? 
In the long run, Google’s entry into the browser 
market is a win-win situation. Microsoft’s devel- 
opment of IE stagnated until Firefox emerged. 
Firefox spurred sleeping Microsoft into action, 
which resulted in a new and improved IE that 
incorporated the best features of Firefox. At the 
very least, Chrome’s arrival on the scene will en- 
courage Microsoft to protect its market position 
by making improvements in IE. This competition 
is good news for users: No matter which browser 
you choose you'll be using a better product. Be- 
yond that, Chrome might be a key step forward in 
the evolution of cloud computing. May the best 
browser win! SOLI 
InstantDoc ID 100324 


Chrome browser. 
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Sqimag.com Community Dialog 


The Wisdom of SQL Mag Readers 


In an 
industry 
where 
information 
is soon 
obsolete, 
Sqlmag.com 
content has 
a long 
shelf-life. 
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New Comments 

on an Old Article 

I’ve mentioned in previous Sqlmag.com Community 
Dialog columns how the reader comments sections on 
some articles take on a life of their own, sometimes for 
long after the original article was published. This tells 
me a couple of things. First, it highlights the staying 
power and long-term usefulness of much of the SOL 
Server Magazine content. Sqlmag.com is a source of 
up-to-date information about SQL Server technology, 
and it's also an archive of thousands of articles, many 
of which contain information that readers find useful 
for years. In an industry where so much information 
becomes obsolete in weeks, months, or even days, it's 
reassuring that Sqlmag.com is a repository for content 
with a long shelf life. 

Here are a couple of cases in point. In “Bulk 
User Create," June 1999, InstantDoc ID 5350, reader 
savindrasingh posted an alternative method for cre- 
ating user accounts in bulk. savindrasingh suggests 
storing the required user information in a text file 
as comma-separated values (CSVs), then provides 
a batch-file For /F command that create the user 
accounts from the data in the text file. You can read 
savindrasingh's complete comment with the command 
in the comments below the article. 


More Shared Wisdom 

Tyler Chessman’s “Using Reporting Services in an 
Internet/Extranet Environment,’ November 2006, 
InstantDoc ID 93830, is another example of an article 
containing helpful reader comments. The latest reader to 
comment online, fsterry, posted his solution to a problem 
he thought other readers might face when using Tyler’s 
solution with Business Intelligence Development Studio 
(BIDS). “Great article... With the Forms Authentica- 
tion Sample implemented, including Tyler’s fixed VB 
file, BIDS could not deploy my work to the server. While 
SSMS seemed fine with the forms authenticated user, 
BIDS did not seem to work with the new extension at 
all.... Solution for me was to develop, and run Deploy, 
of my Report Model from BIDS on a SQL Server 2008 
CTP installation on a different server. Worked on the first 
try.” Keep an eye on the reader comments: You might find 
some nuggets of wisdom from your peers in them! 


DBAs: Whaddya Know? 

The topic of Kalen Delaney’s September 18, 2008, 
SQL Server Magazine UPDATE column, “What Do 
DBAs Need to Know to Do Their Jobs?” is begging 
for answers from readers. Kalen lists a starter set of five 
questions all DBAs should be able to answer, and these 
questions delineate some basic categories of essential 
knowledge for DBAs: database sizing; backing up, 
recovering, and ensuring high-availability of data- 
bases; and database configuration. Kalen also talks 
about DBA jobs sometimes overlapping into other job 
descriptions (development, for example). Should SQL 
Server DBAs “do Windows"—that is, have a basic 
knowledge of Windows server administration? Post an 
answer to Kalen’s question in the reader comments on 
her article, and see how your peers responded. 


Featured Article: 

SQL Server 2008 Review 

SQL Server Magazine has published many articles 
in recent months about SQL Server 2008. What we 
haven't published is a review of the SQL Server 2008 
release; our sister publication, Windows IT Pro, beat us 
to it. Fortunately, SQL Mag subscribers can read the 
entire article, “SQL Server 2008," by Michael Otey at 
www.windowsitpro.com, InstantDoc ID 100315. (The 
article is available free of charge to everyone, as are all 
Windows IT Pro and SQL Server Magazine product 
reviews.) Read the article and tell us whether or not 
you agree with Michael's recommendation. [SQL 


InstantDoc ID 100345 
—Anne Grubb, website strategic editor, 
SQL Server Magazine 


GET ACTIVE ONLINE 
AT SQLMAG.COM! 

Is business intelligence (BI) sans a data warehouse 
better for small-to-midsized businesses (SMBs)? 
Sqlmag.com BI columnist Derek Comingore says 
yes, in his comment on Michelle A. Poolet's article, 
“BI Without the Data Warehouse,” www.sglmag 
.com, InstantDoc ID 100218. Readers, do you 
agree or disagree with Derek? Post a comment and 
share your opinion. 
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SQLScripter 


Automate the generation of database schema scripts 


Kevin Kline 


kevin.kline@ quest.com) is the director of 


technology for SQL Server Solutions at Quest 


Software and a founding board member of 


the international Professional Association for 
SQL Server. He is the author of SQL in a 
Nutshell, 2nd edition (O'Reilly). 


catid=169&entercat=y. 


SQLSCRIPTER 


BENEFITS: You can use SQLScripter to quickly 
and easily automate database schema scripting 


tasks. 


SYSTEM REQUIREMENTS AND NOTES: SQL 
Server 2005; Windows Vista, Windows XP, or 
Windows Server 2000 and later; the Microsoft 


.NET Framework 2.0 


HOW TO GET IT: You can download SQLScripter 
from_www.valinor.co.il/sqlscripter/sq|scripter.html. 


AUTHOR’S NOTE 
We want to hear your feedback on the Tool 
Time discussion forum at sqlforums 
.windowsitpro.com/web/forum/categories.aspx? 


[ 9 going to introduce you to a free tool called 
m SOLScripter, a command-line utility that 


you can use to quickly and easily automate database 
schema scripting tasks, such as script generation for 
disaster recovery or as part of the development pro- 
cess for checking in the database schema. The manual 
process of using the GUI to generate a schema is slow, 
tedious, and prone to mistakes. SQLScripter addresses 
all these problems by transforming database schema 
generation from a slow, manual process into a fast, 
automated one. SQL Scripter improves on SQL Server 
Management Studio’s (SSMS’s) Generate SQL Server 
Scripts Wizard by eliminating the requirement that 
you have to use the GUI to manually build the exact 
composition of the objects you want scripted. 


Using SQLScripter 

SQLScripter was written by Yaniv Etrogi, a SQL 
Server developer and DBA living in Tel Aviv, Israel. 
Yaniv wrote the tool while 
developing a high-avail- 
ability solution for a rep- 
licated environment. The 
subscriber needed to be 
up-to-date with the pub- 
lisher's work so that in the 
event of a disaster the subscriber could take on the 
role of the publisher. 

You can run SQLScripter as a SQL Server Agent 
job step of type "Operating system (CmdExec)" via the 
Windows Task Scheduler, from a batch file, or by manu- 
ally executing the utility from the Windows command 
prompt. You can control the tool via its configuration file, 
SQLScripter.exe.config, 
which uses parameters to 
manage all aspects of the 
schema script generation. 
You can configure the 
utility in a variety of ways. 
For example, you could 
configure SQLScripter to 
be executed on Server! to 
script out a set of objects 
on Server2 and direct the 
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output to a path and file 


on Server3. SQLScripter’s biggest benefit is that it pro- 
vides an easy way for you to automate schema script 
generation from within SSIS jobs. 
SQLScripter uses trusted connections when con- 
necting to local and remote servers. Note that you 
can't use it to script encrypted database objects. Upon 
completion, SQLScripter terminates with a return 
value of 0 on success and 1 on failure. 
You can scope the schema script generation by 
instance, database, and object type. You can also script 
out server-level objects such as jobs and roles. You must 
declare on Output Path, where the script file is stored 
as a local path, mapped drive, network drive, or by its 
Universal Naming Convention name. Of course, the 
account executing SQLScripter must have permission 
to create folders and files at Output Path. To script 
objects, the user must have SELECT permissions or 
otherwise be able to “see” the object to be scripted. You 
can use a .zip file to store the schema generation script 
that’s created and specify a password for the .zip file 
to secure it. 
You can use SQLScripter to perform many different 
tasks, including 
* migrating a database schema between production, 
quality assurance, and development environments 

* keeping the latest schema in source control 

* tracking schema history regularly using an 
automated job 

* creating a script quickly and easily to drop and 
create all indexes for fill factor maintenance and 
defragmentation 

* scripting out important server-level objects before 
the installation of an upgrade, service pack, or 
hotfix to ensure the recoverability of jobs and 
extended stored procedures 


System Configurations 
SQLScripter was developed using SQL Server 2005 and 
uses the Microsoft .NET Framework 2.0, including 
SQL Server Management Objects. It runs on Windows 
Vista, Windows XP, and Windows Server 2000 
and later. It was tested primarily on SQL Server 
2005 in 32-bit environments, but it might work (unsup- 
ported) in 64-bit environments. SQU 
InstantDoc ID 100140 
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As your business intelligence (BI) needs change, fast track 
your development efforts by adopting a single, scalable 
platform based on the trusted technology of Crystal Reports®. 
Reduce your coding requirements with a complete report 
creation, management, distribution and viewing solution 


and experience the ease and confidence that comes with 


choosing one vendor for all your BI deployment needs. 


Choose the flexible BI tools and deployment options you need to get to where you want to be. 
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At SQL Server 
Magazine, we 
believe in bring- 
ing you field-tested, practical 
solutions. Reader to Reader 
content puts experts like 
you directly in contact with 
the solutions other reader- 
experts have found. We're 
proud to feature four Reader 
to Reader solutions this 
month as November's cover 
story. We love receiving your 
practical solutions from the 
field, and | encourage all of 
you to send us your favorite 
solutions to the hard SQL 
Server problems you've 
encountered. 

—Sheila Molnar, 

executive editor, 

SQL Server Magazine 


Ola Hallengren 
DBA at a Nordic investment bank, 


Sweden, ola@ hallengren.com, 
ola hallengren.com 
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Back up databases, check their integrity, 


and optimize indexes 


aintaining databases is an important but 

time-consuming job. I wrote a script, 

MaintenanceSolution.sql, that lets you 
easily perform three common maintenance tasks: 
backing up databases, checking the integrity of data- 
bases, and optimizing indexes. The script is set up so 
that you can call its three main stored procedures— 
DatabaseBackup, IndexOptimize, and Database- 
IntegrityCheck—separately Thus, you can per- 
form each task independently on the databases you 
specify. 

MaintenanceSolution.sql is not only easy to use 
but also offers features such as dynamic index optimi- 
zation. Rebuilding and reorganizing indexes dynami- 
cally has a number of advantages. It lets you rebuild 
or reorganize only the indexes that need to be rebuilt 
or reorganized, so locking and blocking are mini- 
mized and fewer system resources are used. This, in 
turn, can result in higher availability and reduce the 
size of differential and transaction log backups. 

MaintenanceSolution.sql works on the Standard, 
Enterprise, Workgroup, Express and Developer 
Editions of SQL Server 2008 and SQL Server 2005 


SP2 running on X86, X64, or IA64 platforms. The 
solution is supported on the same OSs that SQL 
Server supports. You can download Maintenance- 
Solution.sql by going to www.sglmag.com, entering 
100178 in the InstantDoc ID text box, and 
clicking the 100178.zip hotlink. Aternatively, you can 


download the script at http://blog.ola.hallengren 
.com/blog/ archives/2008/1/1/3440068.html. 


low to Back Up Databases 
DatabaseBackup is the stored procedure in Mainte- 
nanceSolution.sql that performs database backups. 
When you call this stored procedure, it creates a back- 
up directory, performs a database backup, verifies the 
backup file, then deletes any old backup file for each 
database you specify. You specify the database to 
back up and other details using an EXECUTE state- 
ment such as 


EXECUTE dbo.DatabaseBackup 
@Databases = 'USER DATABASES', 
@Directory = 'C:\Backup', 
@BackupType = 'FULL', 
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(Verify = 'Y', 
@CleanupTime = 24 


This statement tells DatabaseBackup to perform a 
full backup of all user databases, verify the backups, 
then delete any backup files older than 24 hours. 

Let’s take a closer look at the EXECUTE state- 
ment. You use the @Databases parameter to specify 
the databases you want to back up. You can select all 
user databases by specifying 'USER_DATABASES' 
or select all system databases by specifying 'SYS- 
TEM_DATABASES'. You can exclude databases 
from the user databases using the syntax 


@Databases = 
"USER_DATABASES, -Databasel, -Database2' 


where -Databasel and -Database2 are the names of 
the databases you want to exclude. With this setup, 
you can, for example, have a backup strategy for one 
database and another backup strategy for all other 
databases. 

Alternatively, you can back up an individual data- 
base by specifying the database’s name, following the 
syntax 


@Databases = 'Databasel' 


To back up two or more individual databases, follow 
the syntax 


@Databases = 'Databasel, Database2' 


Besides the DatabaseBackup stored procedure, the 
IndexOptimize and DatabaseIntegrityCheck stored 
procedures use the @Databases parameter. 

The @BackupType parameter identifies the 
backup type. The stored procedure can perform full 
backups (‘FULL’), differential backups (‘DIFF’), or 
transaction log backups ('LOG) on the databases you 
specify. 

You use the @Directory parameter to specify the 
backup root directory, under which DatabaseBackup 
creates a directory structure consisting of the SQL 
Server instance name, database name, and backup 
type. So, for example, if your SQL Server instance is 
Server1$Instancel and you specify 'AdventureWorks' 
for the (Databases parameter, 'C:\Backup' for the 
@Directory parameter, and FULL' for the @Backup- 
Type parameter, the stored procedure creates the direc- 
tory structure C:\Backup\Server1$Instancel\Adven- 
tureWorks FULL. Each backup files name consists 
of the SQL Server instance name, database name, 
backup type, and date and time of the backup. In this 
example, the filename would be Server!$Instancel_ 
AdventureWorks FULL 20080904 000001.bak. 

The last two parameters are @Verify and 
@CleanupTime. The @Verify parameter controls 
whether the backup file should be verified ('Y") or not 
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CN’). The @CleanupTime parameter specifies when 
to delete old backup files, assuming the backup and 
the verification (if that was selected) was successful. 


How to Optimize Indexes 
Although there aren't any set rules when it comes to 
determining when and how to optimize indexes, there 
are some basic concepts to guide you: 

* Whether an index should be rebuilt, reorganized, 
or left untouched depends on the index's 
fragmentation level. Highly fragmented 
indexes (typically more than 30 percent 
fragmented) should be rebuilt. Indexes 
with little fragmentation (typically less 
than 5 percent fragmented) should be left alone. 
Moderately fragmented indexes (typically between 
5 percent and 30 percent fragmented) should be 
reorganized. (For more information about this con- 
cept, see ^Reorganizing and Rebuilding Indexes" in 
SQL Server 2008 Books Online—BOL—at msdn 
.microsoft.com/en-us/library/ms189858.aspx.) 

* [ndex fragmentation in a very small table has no 
impact on performance. 

* When an index is rebuilt, the statistics are always 
rebuilt and therefore updated. However, when an 
index is reorganized, the statistics aren't updated. 
Therefore, you might want to update the statistics 
after an index is reorganized. 

* An index can be rebuilt online or offline. An offline 
rebuild is faster than an online rebuild. (Note that 
only the Enterprise and Developer Editions of 
SQL Server 2008 and SQL Server 2005 support 
online rebuilds.) 

* When an index contains a large object (LOB) 
column, an online rebuild can't be done. 


The IndexOptimize stored procedure incorpo- 
rates these concepts into its logic. This stored pro- 
cedure categorizes indexes into six groups based on 
their level of fragmentation (high, medium, or low) 
and whether LOB columns are present (LOB) or 
not (NonLOB). The parameters that represent these 
groups are as follows: 

* @FragmentationHigh_LOB represents indexes 
with a high fragmentation level and LOB columns. 

* @FragmentationHigh_NonLOB represents 
indexes with a high fragmentation level and no 
LOB columns. 

* @FragmentationMedium_LOB represents indexes 
with a medium fragmentation level and LOB 
columns. 

* @FragmentationMedium_NonLOB represents 
indexes with a medium fragmentation level and no 
LOB columns. 

* @FragmentationLow_LOB represents indexes with 
a low fragmentation level and LOB columns. 


ORE on the WEB 


Download the code at 
InstantDoc IDs 100178, 
400201, 100213, and 100217. 


Whether you 
should rebuild 
an index, 
reorganize 
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the index’s 
fragmentation 
level. 
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LISTING |: Sample Execute Statement for 
the IndexOptimize Stored Procedure 


EXECUTE dbo.IndexOptimize 

@Databases = 'USER_DATABASES', 

GFragmentationHigh LOB = 'INDEX REBUILD OFFLINE', 
GFragmentationHigh NonLOB = 'INDEX REBUILD ONLINE', 
GFragmentationMedium LOB = 'INDEX REORGANIZE STATISTICS UPDATE', 
GFragmentationMedium NonLOB - 'INDEX REORGANIZE STATISTICS UPDATE', 
GFragmentationLow LOB - 'NOTHING', 

GFragmentationLow NonLOB - 'NOTHING', 

GFragmentationLevell - 5, 

@FragmentationLevel2 = 30, 

@PageCountLevel = 1000 


DateTime: 
Command: ALTER INDEX [IX_Customer_TerritoryID] ON 
[Adventureworks]. [Sales]. [Customer] REORGANIZE 
Comment: IndexType: NonClustered, LOB: No, 
PageCount: 1001, Fragmentation: 6.88235 

Outcome: Succeeded 


DateTime: 


* @FragmentationLow_NonLOB represents indexes 
with a low fragmentation level and no LOB 
columns. 


IndexOptimize uses the avg fragmentation in | 
percent column in the sysdm db index physical 
stats dynamic management view (DMV) to obtain 
the percentage of fragmentation in each index. Using 
the threshold limits you set in the @Fragmentation- 
Levell (lower threshold) and @FragmentationLevel2 
(upper threshold) parameters, it places each index in 
the appropriate group. Indexes with fragmentation 
levels higher than the upper threshold go into one of 
the high fragmentation groups. Indexes with fragmen- 
tation levels at or between the two thresholds go into 
one of the medium fragmentation groups. Indexes 


2008-09-04 11:38:10 


2008-09-04 11:38:21 


Figure | 

Sample log entry for 
the IndexOptimize 
stored procedure's 
ALTER INDEX 
command 
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with fragmentation levels under the lower threshold 
go into one of the low fragmentation groups. 
IndexOptimize can also place indexes in one of 
the low fragmentation groups based on their page 
count. Indexes under the size specified in the @Page- 

CountLevel parameter go into one of the low frag- 

mentation groups. 

For each group, you can select one of the follow- 
ing actions: 

*' INDEX REBUILD ONLINE'— Tells the stored 
procedure to rebuild the indexes online. (You need 
to be running the Enterprise or Developer Edition 
of SQL Server 2008 or SQL Server 2005 to use this 
option.) 

*' INDEX REBUILD OFFLINE — Tells the stored 
procedure to rebuild the indexes offline. 

*' INDEX. REORGANIZE' —Tells the stored proce- 
dure to reorganize the indexes. 

*' INDEX REORGANIZE STATISTICS 
UPDATE — Tells the stored procedure to reorga- 


nize the indexes and update the statistics. 
*'STATISTICS UPDATE — Tells the stored proce- 
dure to update the statistics. 
*'NOTHING — Tells the stored procedure to do 
nothing to the indexes. 


So, for example, the EXECUTE statement in List- 
ing 1 tells IndexOptimize to rebuild indexes that are 
more than 30 percent fragmented, online if possible 
(no LOBs). If these highly fragmented indexes have 
LOBs, an offline rebuild is to be done. Indexes with a 
fragmentation level between 5 percent and 30 percent 
are to be reorganized and have their statistics updat- 
ed. Indexes that are less than 5 percent fragmented 
or have fewer than 1,000 pages aren't to be touched. 
IndexOptimize uses T-SQL’s ALTER INDEX com- 
mand to rebuild and reorganize indexes. 


How to Check Databases’ 
Integrity 

The DatabaseIntegrityCheck stored procedure uses 
T-SQL’s DBCC CHECKDB command to perform 
integrity checks. Using this stored procedure instead 
of the Database Maintenance Plan Wizard's Check 
Database Integrity Task might mean that you don't 
have to install the hotfix for bug 50001012. (The 
Check Database Integrity Task can lose database con- 
text under certain circumstances in SQL Server 2005 
builds 3042 through 3158—see support.microsoft 


.com/kb/934458.) 


The EXECUTE statement you use to run 
DatabaseIntegrityCheck is simple. You just need to 
specify the databases you want to check. For example, 
the statement 


EXECUTE dbo.DatabaseIntegrityCheck 
(Databases = 'USER DATABASES' 


checks the integrity of all user databases. 


All Types of Information Are 
Readily Available 

The DatabaseBackup, IndexOptimize, and Database- 
IntegrityCheck stored procedures have thorough log- 
ging and error handling. The start and end time, com- 
mand text, and output are logged for each command 
in the stored procedures. Additional information is 
logged for the IndexOptimizes ALTER INDEX com- 
mand, as Figure | shows. All command information is 
immediately written to a log file. You can find informa- 
tion about how the stored procedures handle errors in 
MaintenanceSolution.sql’s documentation at blog.ola 


.hallengren.com/ attachments/3440068/Document 
ation.html. The documentation also includes informa- 


tion about how to use each stored procedure as well as 
answers to frequently asked questions. 
InstantDoc ID 100178 
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SQL SERVER SOLUTIONS FROM THE FIELD 


MODIFY Words (and Not Strings) 
in Tables 


UDF changes only what it’s supposed to 


few weeks ago I searched many SQL Server 
A for T-SQL code that modified 

words. Specifically, I needed to replace 
a word in a table's text column. I found many code 
samples, but in all cases the code modified a string 
rather than a word. As a result, if the word I wanted 
to modify was part of another word, the modifica- 
tion also changed this word. For example, changing 
the word Bikes to Trucks using these code samples 
also modified SuperBikes to SuperTrucks. In many 
cases, the string-modification code also didn't catch 
the word I was searching for if the word was at the 
beginning or end of the text column. 

Because I didn’t find what I was looking for, I 
wrote a user-defined function (UDF) named cm 
ModifyWord. You can use this UDF to modify a 
word (and not a string) in any position within a text 
column (i.e., beginning, middle, or end). The UDF 
even works when words are delimited by punctuation 
characters such as a period (.), comma (,), semicolon 
(G), colon (:), or exclamation point (!). 

You can download cm_ModifyWord by going to 
www.sqimag.com, entering 100217 in the InstantDoc 


TRACK Database 
Disk-Space Usage 
on a Granular Level 


Find out how much space is 
being used by tables and their indexes 


ne of the housekeeping challenges DBAs 
Q face is to keep track of their databases’ 

growth and how much space is being 
used by each table and index in those databases. To 
meet this challenge, I wrote a stored procedure, usp 
SpaceUsedAnalyzer, that extends the functionality of 
the sp. spaceused system stored procedure to present 
more detailed information. 
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Christian Mercure 


ID text box, and clicking the 100217.zip hotlink. This 
UDF works on SQL Server 2005. 

The code in Listing 2 demonstrates how to use 
the cm. ModifyWord function. This code replaces the 
word Bikes with Trucks in the Name column of the 
AdventureWorks database's Sales.Store table. If you 
run this code, you'll find that it doesn't change Super- 
Bikes to Super Trucks. 
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LISTING 2: Code that Uses cm ModifyWord to 
Replace the Word Bikes with Trucks 


USE [AdventureWorks] 
DECLARE QGOldWord AS Varchar(255) 
DECLARE @NewWord AS Varchar(255) 
SET G0ldWord-'Bikes' 
SET @Neword='Trucks' 


SELECT [Name],len(dbo.cm ModifyWord([Name], GOldWord, GNewWord, '. 


FROM [Sales].[Store] 
WHERE [Name] LIKE '%'+@01dWord+'%' 


UPDATE [Sales].[Store] 
SET [Name]-dbo.cm ModifyWord([Name], @OldWord, GNewWord, '.,:;!') 
WHERE [Name] Like '%'+@0]ldWord+'%' 


SELECT [Name] 
FROM [Sales].[Store] 
WHERE [Name] Like '%'+@NewWword+'%' 


Shaunt 
Khaldtiance 


The usp SpaceUsedAnalyzer stored procedure is 
particularly helpful for finding out how much space is 
being used by indexes. As Figure 2, page 18, shows, you 
can use the stored procedure to reveal such details as 
how much space is being used by clustered and non- 
clustered indexes and their type of space allocation. 

You can download usp SpaceUsedAnalyzer by 
going to www.sglmag.com, entering 100213 in the 
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Table Name 
ExternalAccessLog dbo 
ExternaldccessLog dbo 


Figure 2 


Sample output from the 
usp_SpaceUsedAnalyzer 
stored procedure 


Schema Name 


Index Name 
2679283 
423949 


PK_ExternalAccessLog 1 
PK Esternal&ccessLog 1 


InstantDoc ID text box, and clicking the 100213 
Zip hotlink. You simply run this stored procedure on 
the SQL Server 2008 or SQL Server 2005 machine 
for which you want to evaluate database disk-space 
usage. It won't work with SQL Server 2000 or earlier 
because it uses system tables that are available only in 
SQL Server 2005 and later. 

As the following syntax shows, usp_SpaceUsed- 
Analyzer takes two parameters: 


EXEC usp SpaceUsedAnalyzer 
{'summary'|'details'}, 
{'1 [desc|asc][,2 [desc|asc],...n]'} 


You use the first parameter to specify whether you 

want to display data at the table level (‘summary’) or 

the index level (‘details'). You use the second param- 

eter to specify how you want that data sorted. When 

you use 'summary' for the first parameter, you can 

sort the returned data by the following columns in 

descending (desc) or ascending (asc) order: 

* Schema. This column, which is represented by the 
value 1, identifies the schema for each table. 

* Table Name. This column, which is represented by 
2, specifies the name of each table. 

* Number of Rows. This column, which is represent- 
ed by 3, notes how many rows are in each table. 

* Reserved Space (MB). This column, which is rep- 

resented by 4, shows 
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coverage this month. Drop us an 
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the amount of space 
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solutions. Send them to ' dexSize (MB). 
This column, which 
R2R@sqlmag.com. If we print your is represented by 6, 
submission, you'll get $100. specifies the total 
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amount of space 
used by all the indexes in each table. 
* Unused Space (MB). This column, which is repre- 
sented by 7, reveals the amount of available space 
in each table. 


When you use 'details' for the first parameter, you can 


sort the returned data by the following columns in 


IndexID No. Pages index type desc 
CLUSTERED INDEX 
CLUSTERED INDEX 


alloc. unit type desc Used Space (MB) 
LOB DATA 20931.8984375 
IN_ROW_DATA 3312.1015625 


descending or ascending order: 

* Table Name. This column, which is represented by 
l, specifies the name of each table. 

* Schema Name. This column, which is represented 
by 2, identifies the schema for each table. 

* Index Name. This column, which is represented by 
3, provides the name of each index in each table. 

* Index ID. This column, which is represented by 4, 
contains the ID of each index. 

* No. Pages. This column, which is represented by 5, 
specifies the number of pages in each index. 

* index type desc. This column, which is repre- 
sented by 6, notes whether each index is clustered 
or nonclustered. 

e alloc unit. type desc. This column, which is repre- 
sented by 7, identifies the type of space allocation 
for each index. 

* Used Space (MB). This column, which is represent- 
ed by 8, reveals how much space each index is using. 


For example, the statement 


EXEC usp SpaceUsedAnalyzer 
'summary','6 desc' 


displays table-level data that's sorted in descending 
order by the total amount of space used by each ta- 
ble's indexes. If you want to find the tables with the 
most data, you'd run the query 


EXEC usp SpaceUsedAnalyzer 
'summary','5 desc' 


The following query highlights the nonclustered in- 
dexes that take the most amount of space: 


EXEC usp SpaceUsedAnalyzer 
'details','6 desc,8 desc' 


As you can see, the usp SpaceUsedAnalyzer 
stored procedure provides the disk-space usage de- 
tails you need to manage disk space more efficiently. 
For instance, you can use usp SpaceUsedAnalyzer to 
identify indexes that are taking a lot of space. If any 
of those indexes aren't being used, you can remove 
them to free up some space. (See “Evaluate Index 
Usage in Databases," October 2008, InstantDoc ID 
99985, for a stored procedure you can use to find in- 
dexes that aren't being used.) 

InstantDoc ID 100213 
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Quickly FIND 


Your Worst-Performing 
T-SQL Statements 


Stored procedure lets you avoid the 
hassles associated with writing DMV and 


DMF queries on the fly 


Server 2008 and SQL Server 
2005 include dynamic man- 
agement views (DMVs) and 


dynamic management functions (DMFs) that provide 
performance-related information. However, the most 
useful information is obtained when you combine 
results from various DMVs and DMFs. For exam- 
ple, you can combine their results to identify T-SQL 
statements that are using an excessive amount of 
server resources. 

When you want to quickly identify resource- 
intensive T-SQL statements, you don’t want to 
have to wade through DMV and DMF syntax 
documentation or search for information on how to 
perform JOIN operations. To avoid the hassles 
associated with writing DMV and DMF queries 
on the fly I wrote a parameter-driven stored 
procedure named usp_Worst_TSQL. You can down- 


load usp Worst TSQL by going to www.sqlmag 


..com, entering 100201 in the InstantDoc ID text box, 
and clicking the 100201.zip hotlink. 

The usp Worst. TSQL stored procedure executes 
three DMVs (sys.dm exec query. stats, sys.dm_exec_ 
cached plans, and sys.dm exec sql text) and one 
DMF (sys.dm exec plan attributes), then joins and 
manipulates the DMV and DMF results to provide a 
variety of statistics. Table 1, page 20, shows some of 
the available statistics. (The 100201.zip file contains 
the entire list in the file StatisticsReturned.xls.) With 
these statistics, you can identify the worst-performing 
T-SQL statements. 


How to Use usp Worst TSQL 
The usp Worst TSQL stored procedure has three 
optional parameters—@DBNAME, @COUNT, and 
@ORDERBY—that let you restrict and sort the 
stored procedure’s output. The @DBNAME param- 
eter lets you retrieve T-SQL statements for a single 
database. If you don't include this parameter, the 
stored procedure returns the statements for all the da- 
tabases on the server. 
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The @COUNT parameter limits the number 
of rows (Le, T-SQL statements) returned by usp_ 
Worst TSQL. If you use this parameter, the stored 
procedure returns only the top x rows (where x is 
equal to the @COUNT value). If you don't use this 
parameter, the stored procedure returns all rows. 

The @ORDERBY parameter identifies the statis- 
tic to sort the rows by. Table 1 shows some of the val- 
ues you can use with this parameter. No matter which 
statistic you choose, the rows are sorted in descending 
order. If you don't include the ()ORDER BY param- 
eter, the stored procedure sorts by average I/O usage. 

With the (2DBNAME, @COUNT, and 
@ORDERBY parameters, you can quickly generate 
customized worst-performance reports. For example, 
the call 


EXEC Examples.dbo.usp Worst TSQL 
@DBNAME='AdventureWorks', @COUNT=6, 
@ORDERBY='ACPU' ; 


returns the top six T-SQL statements that have the 
highest average CPU usage in the AdventureWorks 
database. The call 


EXEC Examples.dbo.usp Worst TSQL 
@DBNAME='AdventureWorks', @COUNT=5; 


returns the top five T-SQL statements that have the 
highest average I/O usage values in the Adventure- 
Works database. Figure 3, page 20, shows an excerpt 
of sample output from this query. (Only the first seven 
columns are displayed, and the last T-SQL statement 
is truncated.) As you can see, the T-SQL statement in 
the first row is by far the worst-performing in terms 
of average I/O. 

In Figure 3, notice that some of the database 
names have an asterisk (*) after them. An asterisk 
indicates that the dbid column in the sys.db_exec_ 
plan_attributes DMV was used to identify the data- 
base name for that statement. When this occurs, the 
database name is the database context that was used 
when the statement was executed, so it might not 


Gregory A. Larsen 


Lead DBA, Washington State Department of 


Health, Tumwater, Washington, and part-time 


consultant, gregalarsen @ msn.com, 


www.sqlserverexamples.com 


November 2008 


19 


EE ee SQL SERVER SOLUTIONS FROM THE FIELD 


Object Name 


Database Name 


Adventur ewor ks * 
Adventur ewor ks * 
Adventur ewor ks * 


Adventur ewor ks * 
Adventur ewor ks 


Statement 


select * from Sales.vIndividualDemographi cs 


select * from Person.vAdditionalContactInfo 
select distinct(SalesOrderID) from 
Sales.SalesOrderDetail 
select count(*) from Production. Product 
WITH [BOM_cte] ([ProductAssemblyID], 
[ComponentID], [cC 
[PerAssemblyQty] , 


NULL 
NULL 
NULL 
where ProductID = 711 

NULL 


nent Desc], 
StandardCost], [ListPrice], 


[BOMLevel], [RecursionLevel]) -- CTE name and 


Figure 3 


Excerpt from sample 
usp Worst TSOL 


output 


reflect the actual database name if three-part naming 
conventions are used. 

If you run usp Worst. TSQL with no parameters, 
such as 


EXEC Examples.dbo.usp Worst TSQL 


the stored procedure returns all the T-SQL statements 
(sorted by average I/O usage) ran against all the data- 
bases on the server. These results can help identify the 
databases against which the most resource-intensive 
statements are being run. 

I use usp Worst. TSQL frequently and have en- 
countered quirky results only one time. When I ran 
the stored procedure against a SQL Server 2005 
instance running on my Windows XP laptop, it re- 
turned negative numbers for some of the columns. I 
believe this occurred because the sys.dm exec query . 
stats DMV's total worker time and total elapsed - 
time statistics contained negative numbers, which is 
related to SQL Server not being able to handle differ- 


uspGetBillof 
Materials 


ent CPU frequencies, as outlined in the Microsoft ar- 
ticle "SQL Server timing values may be incorrect when 
you use utilities or technologies that change CPU fre- 
quencies" (support.microsoft.com/kb/931279). This 
problem didn't occur when I ran usp Worst. TSQL 
against a SQL Server 2008 instance on the same lap- 
top or against any of my SQL Server 2005 instances 
running on Windows Server 2003 servers. 


A Handy Report 
The usp Worst TSQL stored procedure lets you 
quickly produce customized reports showing the 
poorest-performing T-SQL statements—all without 
having to remember or search through documenta- 
tion on how to perform DMV, DMF, and JOIN op- 
erations. You can then share this information with the 
people who wrote the T-SQL statements (e.g., DBAs, 
programmers) so that they can fine-tune their T-SQL 
statements. E 
InstantDoc ID 100201 


TABLE |: Sample of the Information Returned by usp Worst TSQL 


Database Name 


Name of the database against which the T-SQL statement was run or the context in 


which the statement was run if the database name is followed by an asterisk. 


Schema Name The schema in which object is associated. Is NULL for adhoc or prepared SQL statements. 


Cached Plan Object type as identified by the sys.dm exec cached plans DMV. 
objtype 


Average lOs The T-SQL statement's average I/0 usage. Calculated with the formula (total numberof ^ NO 


logical reads + total number of logical writes + total number of physical reads)/execution 


count. 


Avg Logical Reads Average number of logical reads performed by the T-SQL statement. Calculated with the ALR 


formula total number of logical reads/execution count. 
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Turn Business 
Intelligence Into 
Business Advantage 


ALTERNATIVE THINKING ABOUT BUSINESS INTELLIGENCE: 


Alternative thinking isn't just about access to statistics and 
information. It's about delivering the insights you need to make 
intelligent decisions that differentiate your company from 
everyone else. 


It’s about relying on HP ProLiant and Integrity systems to run 
SQL® Server™ 2008, so you can focus your efforts on business 
innovation. (And how cool is that?) 


It’s about working in close partnership with HP to instantly turn 
your most relevant information into informed business 
decisions. (And look oh-so-smart in the process.) 


It’s about experiencing a change-ready BI infrastructure that 
delivers actionable insight wherever it needs to be - turning 
your company into a competitive leader. (And what's not to 


love about that?) 


Technology for better business outcomes. 


HP Business Intelligence (BI) 

* Over 2,000 BI solutions for Fortune 500 and Global 2000 customers 

* Microsoft's SQL Accelerator for Bl was developed on HP ProLiant platforms 
* Almost 50% of SQL Server users build their Bl on HP Servers 


www.hp.com/solutions/microsoft/sqlbi 


Microsoft and SQL Server are registered trademarks of Microsoft Corporation in the United States and/or other countries. 
©2008 Hewlett-Packard Development Company, LP. 
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Without a doubt, Microsoft is one of the most 
powerful forces in technology today, and 
everyone seems to have an opinion about 
what the Redmond giant does. What's yours? 


e 2008 ITTV “If | Ran Microsoft" 
>, Video Contest is your chance 
N to tell it like it Is. 


The first 250 entrants earn a free T-shirt just for 
s- participating (one per video), plus are entered 
0 Win one of three 8GB Zunes to be given away. 


* 


Visit www.ittv.net today 
Aul video contest rules and all the details. 
— Or stop by our booth at Connections in 

Las Vegas, Nov. 11-13, 2008, to submit 
your.entry. Contest ends Dec. 31, 2008. 
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Calculate 


ps and percentile ranks are values 
used in statistical analysis of data. Given 
a set of values representing certain observations 
or scores (e.g, students’ marks in an exam), 
a percentile is a value below which a certain 
percent of the scores fall. For example, for the 
following exam scores: 40, 55, 70, 70, 70, 75, 
75, 80, 80, 95, and 100, the 50th percentile (also 
known as the median) is the mark below which 
50 percent of the scores fall (in this case, 75). The 
25th percentile (also known as the first quartile) 
is 70, and so on. 

A percentile rank is the percent of values in 
a certain set of observations or scores that are 
lower than a certain value. For example, using 
the same scores, the percentile rank of the score 
75 is 50 percent because 50 percent of the scores 
are lower than 75. The percentile rank of the 
score 80 is 70 percent, and so on. 

After I explain how to calculate percentile 
ranks using T-SQL, Ill show you how to calcu- 
late percentiles. In my examples I’ll use a table 
called Marks. To follow my demonstration, 
you'll need to run the code in Listing 1 to create 
the Marks table in the tempdb database and 
populate it with sample data. 

Note that no standard algorithm exists for 
calculating percentiles. I use algorithms similar 
to the ones used by Microsoft Excel. If you want 
to implement other algorithms, you'll probably 
need to adjust my techniques. 


Percentile Ranks 

As I already mentioned, a percentile rank is the 
percent of values from a certain data set (call it 
S) that are smaller than a certain value (call it v). 
The PERCENTRANK(S, v) function in Excel 
uses different calculations based on whether the 
given value appears in the given data set. Ifit does, 
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Percentiles 


Use I-SQL to improve your statistical analysis 


FEATURE 


the function uses the following formula: (rnk-1)/ 
(cnt-1), where rnk represents the rank of the value 
(number of values in S that are smaller than v, plus 
1) and cnt represents the count of values in S. 

The implementation of the percentile rank 
calculation for a value that appears in the set 1s 
simple; all you need is the rank of the value and 
the count of the values in the set. The rank can 
be calculated in T-SQL using the RANK func- 
tion, and the count can be calculated using the 
COUNT function. The code in Listing 2, page 
24, calculates the percentile rank of each mark 
from the Marks table, generating the output 
shown in Table 1, page 24. 

The query that defines the common table 
expression (CTE) called MarksRnkCnt 
calculates, for each mark, the rank of 
themark (column rnk) usingthe RANK 
function and the COUNT of rows in the 
table (column cnt) using the COUNT 
function with the OVER clause. The outer 
query implements the formula for percentile 


Itzik Ben-Gan 


(IBen-Gan @ SolidQ.com), a mentor 

at Solid Quality Learning, teaches, lectures, 
and consults internationally. He manages the 
Israeli SQL Server Users Group, is a 

SQL Server MVP, and is the author of 

the Inside Microsoft SQL Server 2005: 

T-SQL series (Microsoft Press, 2006). 


ORE on the WEB 


Download the listings at 
InstantDoc ID 100129. 


LISTING I: Script to Create and Populate 
the Marks Table 


SET NOCOUNT ON; 
USE tempdb; 


IF OBJECT ID('dbo.Marks', 'U') IS NOT NULL DROP TABLE dbo.Marks; 
GO 


CREATE TABLE dbo.Marks 

(c 

studentid VARCHAR(10) NOT NULL PRIMARY KEY, 
mark INT NOT NULL 

3 


INSERT INTO dbo.Marks(studentid, mar 
INSERT INTO dbo.Marks(studentid, mar 
INSERT INTO dbo.Marks(studentid, mar 
INSERT INTO dbo.Marks(studentid, mark) VALUES('D', 80); 
INSERT INTO dbo.Marks(studentid, mark) VALUES('E', 79); 


) VALUESC'A', 75); 
) 
) 
INSERT INTO dbo.Marks(studentid, mark) VALUES('F', 75); 
) 
» 
) 
) 
J) 


VALUES('B', 79); 
VALUES('C', 79); 


INSERT INTO dbo.Marks(studentid, mark) VALUES('G', 109); 
VALUES('H', 55); 
VALUES('I', 95); 
VALUES('J', 89); 
VALUES('K', 49); 


INSERT INTO dbo.Marks(studentid, mar 
INSERT INTO dbo.Marks(studentid, mar 
INSERT INTO dbo.Marks(studentid, mar 
INSERT INTO dbo.Marks(studentid, mar 
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CALCULATE PERCENTILES 


LISTING 2: Query 


Calculating Percentile Ranks 


WITH MarksRnkCnt AS 


SELECT studentid, mark, 
RANK() OVER(ORDER BY m 
COUNT(*) OVERQ AS cnt 

FROM dbo.Marks 


) 

SELECT studentid, mark, 
1.*(rnk-1)/(cnt-1) AS pc 

FROM MarksRnkCnt 

ORDER BY mark; 


TABLE |: Output of Query 


in Listing 2 


studentid mark 


con — fo — IEEE n Ni Co foo — Ee 
~ 
5 


Q Editors Note 


Thanks for your 
feedback about 
running listings 
and tables in print versus 
online only. We're trying 
to include more listings 
and tables in print, as 
space permits. 

—Lavon Peters, 

senior editor 


24 November 2008 


rank using the expression 
1*(rnk-1)cnt-1). The use of 
1. (one dot) here is to cause 


a LS mak, implicit conversion of the 


LISTING 4: Query Calculatin 


Percentile Rank for any Single Value 


DECLARE @mark AS INT; 
SET @mark = 85; 


operands of the expression 
to numeric values, otherwise 
T-SQL would use integer 
division between the integer 
operands. 

If the given value doesn’t 
appear in the set, the PERCENTRANK 
function in Excel interpolates to return the 
result. In such a case, the formula the func- 
tion uses is: 


trnk 


pctrnk, smaller val 

+ (v - smaller val) / 
(larger val - smaller val) 
* (pctrnk larger val 

- petrnk, smaller val) 


percentilerank 


0.00000000000 where v represents the 
0.10000000000 given value, smaller_val 
0.20000000000 and larger_val represent 
0.20000000000 the two closest values 
0.20000000000 from the data set that are 

smaller and larger than 
0/50000000000 v, and pctrnk larger val 
0.50000000000 and pctrnk smaller val 
0.70000000000 represent the percentile ranks 
0.70000000000 of those two values. As an 
0.90000000000 example, 85 1s à mark that 

doesn't appear in the Marks 
100000000000 


table. The two closest marks 
smaller and larger than 85 are 
80 and 95, and their percentile ranks are 0.7 and 
0.9 (see Table 1). The percentile rank of the mark 
85 1s therefore calculated as: 


LISTING 3: Query Calculating 
Percentile Rank Ranges 


WITH MarksRnkCnt AS 
C 


SELECT studentid, mark, 
RANK() OVER(ORDER BY mark) AS rnk, 
DENSE RANK() OVERCORDER BY mark) AS drnk, 
COUNT(*) OVERO AS cnt 

FROM dbo.Marks 


), 
PctRanks AS 
C 


SELECT DISTINCT 

drnk AS rownum, 

mark, 

1.*(rnk-1)/(cnt-1) AS pctrnk 
FROM MarksRnkCnt 


) 
SELECT 
Cur.rownum, 
Cur.mark AS mark from, Nxt.mark AS mark to, 
Cur.pctrnk AS pctrnk from, Nxt.pctrnk AS 
pctrnk to 
FROM PctRanks AS Cur 
JOIN PctRanks AS Nxt 
ON Nxt.rownum = Cur.rownum + 1; 


WITH MarksRnkCnt AS 
K 


SELECT studentid, mark, 
RANK() OVERCORDER BY mark) AS rnk, 
DENSE RANK() OVERCORDER BY mark) AS drnk, 
COUNT(*) OVERO AS cnt 

FROM dbo.Marks 


), 
PctRanks AS 


(t 
SELECT DISTINCT 


drnk AS rownum, 

mark, 

1.*(rnk-1)/(cnt-1) AS pctrnk 
FROM MarksRnkCnt 


), 
PctRankRanges AS 
¢ 


SELECT 
Cur. rownum, 
Cur.mark AS mark_from, Nxt.mark AS mark_to, 
Cur.pctrnk AS pctrnk from, Nxt.pctrnk AS pctrnk to 
FROM PctRanks AS Cur 
JOIN PctRanks AS Nxt 
ON Nxt.rownum = Cur.rownum + 1 


) 
SELECT 


CASE @mark 
WHEN mark from THEN pctrnk from 
WHEN mark to THEN pctrnk to 
ELSE 
pctrnk from 
+ 1.*(@mark - mark from) 
/ (mark to - mark from) 
* (pctrnk to - pctrnk from) 
END AS pctrnk 


FROM PctRankRanges 
WHERE (@mark > mark from AND @mark <= mark to) 


OR Crownum = 1 AND @mark = mark from); 


0.7 + (85 - 80) / (95 - 80) * (0.9 - 0.7) = 0.766667 


This means that with interpolation, 76.6667 per- 
cent of the marks are lower than the mark 85. 

To add the interpolation logic to the calcula- 
tion, you first need to pair adjacent marks from 
the Marks table, and calculate their percentile 
ranks. This 1s achieved with the code in List- 
ing 3, generating the output shown in Table 2. 

The code that defines the CTE called Marks 
RnkCnt calculates a rank, a dense rank, and 
a count for each mark from the Marks table. 
The code that defines the CTE called PctRanks 
queries MarksRnkCnt, calculates the percentile 
ranks as I previously explained, and removes 
duplicate rows. The dense rank becomes a row 
number after the removal of duplicates. The 
outer query joins two instances of PctRanks 
aliased as Cur and Nxt to pair adjacent marks 
and their respective percentile ranks. 

The code in Listing 4 shows how to add logic 
that for an input value (stored in the variable 
@mark) calculates the percentile rank of the 
input. The code defines a CTE called PctRank 
Ranges based on the last query from List- 
ing 3. The outer query in Listing 4 identifies 
the relevant pair of marks and percentile ranks 
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the migration to SQL Server 2008 
begins, the inevitable question will 
be asked in many environments: 
when we deploy SQL Server 
2008, how can we ensure it 
is highly performing? This guide will address the 
factors and challenges involved with implementing 
high performance SQL Server 2008 instances and 
databases. 


How to Size Servers 

It must be stated up front that the way to scale SQL 
Server is up, not out. This has been a sore point for 
many IT shops accustomed to products where it is 
easy just to throw another server at the problem. That 
does not really solve any issues: it is often a patch 

for an underlying problem that is ignored. You can 
scale out with SQL Server, but it takes planning when 
the application is developed. All methods described 

in this guide are related to scale up. The three main 
components related to server sizing are obvious from 
a workload and performance standpoint: processor, 
memory, and disk. When considering your workloads, 
keep in mind if you are going to be doing OLTP, OLAP 
or DSS. Each has its own signature. 


The most crucial factor for the performance of any 
SQL Server deployment is the disk subsystem since 
disk I/O is often slower than accessing memory or 
processor. Storage is usually centralized in many IT 
environments, giving administrators and DBAs little 
control over its configuration. Choosing what type 

of RAID may not be an option, so knowing how the 
storage is configured and working well with the group 
who owns and administers the storage is crucial. 
While one should never place heavily used data and 
log on the same disks. This is easier said than done, 
because in the central storage model of the IT world, 
even if those files are placed on two disks presented 
to a server, on the backend, the underlying storage of 
those disks may actually be sitting right next to each 
other. Just because it is on a SAN does not mean it 
will necessarily be highly performing. If you are going 
to be deploying high performance, mission-critical 
SQL Server, it is always recommended that a dedicated 
SAN, or at the very least, total control over the disk 
configuration at the SAN level is granted to the team 
deploying the databases. Otherwise, there is a good 
chance that the disk performance may not meet the 
needs for the application. 


Sizing storage for SQL Server comes in two 

flavors: throughput (measured in I/Os) and overall 
storage capacity needs. I/O is more important for 
performance: without the proper I/O bandwidth 
with low I/O latency, SQL Server will always perform 


poorly. I/O is reads and writes, not one or the other. 
Performance is not only important for data and log, 
but also tempdb if it is used heavily. When configuring 
your storage for SQL Server, make sure that you check 
with the vendor to see if they recommend alignment 
with DISKPART in Windows and get their offset. If 

the vendor recommends disk alignment and you do 
not do it, you will have two physical I/Os for every 
logical I/O issued from Windows up through Windows 
Server 2003. Windows Server 2008 accounts for 

this alignment automatically. Disks used with SQL 
Server should be formatted with the proper stripe size 
(usually 64k to match its largest I/O operation — the 
readahead); never assume a default stripe size is right 
— check with your Windows administrators before 
installing SQL Server. 


How much storage you need in terms of size is 
dependent upon many factors, not the least of 
which is how many years the system will be deployed 
and regulations which require longer periods of 

data retention. See the section " Growth" for more 
information. Designing the storage and file layout 

is much easier for SQL Server once I/O and storage 
sizing is solved. Remember to account for storage 
and performance needed for backups as well. To get 
better read performance you add more physical disks 
and to improve write performance, you add additional 
controller channels. Ensuring the HBA card settings 
are correct also helps. 


Memory is the next most important factor in sizing 
a server for SQL Server. Objects, users, network 
connections — each consumes memory in addition 
to what SQL Server actually uses for processing, 
transformations, and cache. Cache is the most 
important aspect because if the data is in memory, 
it reduces the need to go out to disk to read the 
information. To see how much memory each object 
uses, consult the SQL Server Books Online topic 
"Memory Used by SQL Server Object Specifications". 
The calculations provided will prove invaluable in 
helping to size your SQL Server implementations. 


Historically, processor is third on the list when it 
comes to performance bottlenecks in SQL Server. 
There are plenty of applications that have high CPU 
utilization, but most hit the wall when it comes to 
disk and memory way before you worry about slices 
of CPU. When it comes to sizing how much capacity 
and numbers of processors you need, remember to 
factor in any other processes that will be running on 
that server including utilities like third-party backup 
compression software which adds to CPU utilization. 
You can also do much more now with two, four, six, 
and most likely in the near future, eight cores per 
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Gary Erickson, Unisys vice president of the Microsoft 
Global Alliance, talks about the high-value relationship 
Unisys and Microsoft together offer to customers. 


How do Unisys and Microsoft 
work together? 


Erickson: The Unisys and Microsoft 
Solutions Alliance delivers highly secure, 
scalable solutions that harness the 
power of people and technology so that 
our customers can achieve a Real-Time 
Infrastructure. We help our customers 
achieve total alignment of their business 
strategy, process, applications, and 
infrastructure. Unlike other companies that 
deliver on isolated technology projects, we 
help our clients see and understand all the 
technology influences and interactions that 
affect their business operations. Our joint 
solutions address high-priority, high-return 
customer needs in the areas of: 
* Managed services 
e Virtualization 
* Core infrastructure optimization 
e Systems Management and Automation 
e High-Performance SQL Server and BI 
* Unified Communications 
e Net/SOA 
* Collaboration & Enterprise Content 
Management 
Together, we provide our customers with 
proven industry solutions, the visibility to 
align business and technology, and dedi- 
cated expertise for delivery assurance. 


How long have Microsoft and Unisys 
had an alliance? 


Erickson: The history of our alliance 

began well over 10 years ago. During this 

time, Unisys and Microsoft accomplished 

a number of outstanding technology 

milestones around Windows and SQL Server 

including: 

* Engineering collaboration on Windows 
Server Datacenter edition....and 


* Record setting benchmarks for SQL Server 
Transaction Processing and SQL Server 
Analysis Services 

* In fact, Unisys provides one of the world's 
fastest performing database platforms 
with Windows Server 2008 and SQL Server 
2008. Record-setting benchmarks include: 

* SQL Server 2008 Integration Services 
package load times (1TB in « 30 
minutes) 

* Microsoft Dynamics CRM 4.0 sub- 
second response times with SQL 
Server 2008 


Can you tell me something about 
Unisys key Microsoft-based 
solutions? 


Erickson: The core of our Microsoft offer- 
ings and solutions are based on Windows Ser- 
ver 2008. Unisys relies on Windows Server 
2008 to provide a scalable operating system 
platform for Unisys Enterprise-class servers. 
Unisys is leveraging its extensive 
enterprise server expertise to provide new 
robust and cost- effective solutions based 
on Windows Server 2008 virtualization 
technology for large-scale desktop and 
Exchange environments. This commitment 
to Windows Server allows us to continue our 
technology and services leadership in server 
consolidation and datacenter rationalization. 


Why do you think your alliance with 
Microsoft works so well? 


Erickson: Unisys' ability to deliver innovative 
and powerful solutions on the Microsoft 
platform connects the value of IT to our 
customers' business needs. Our alliance 
works because we're able to focus on what's 
important to our customers, get the job 
done and deliver on results. 


For more information about Unisys solutions built on the Microsoft platform, 
please visit www.unisys.com 


processor than you ever could with older, slower single- 
core systems. Modern applications tend to be designed 
to scale much better in this symmetrical multiprocessing 
(SMP) environment. Keep in mind that there is more to 
performance than cores — you do need to worry about 
the performance of a single core. More cores does not 
necessarily mean better performance. Learn about the 
architecture you are going to be implementing. 


The bigger question these days is the following: 32-bit 

or 64-bit? At this point, assuming you are planning on 
deploying dedicated SQL Server instances on the intended 
hardware with no other applications and there are no 
application constraints (such as 32-bit extended stored 
procedures that are incompatible with 64-bit), go with 
the x64 architecture. Older 64-bit processors in some 
cases were slower in terms of clock speed than their 32- 
bit counterparts, but the speeds have now caught up. 
64-bit will not necessarily double your computing power; 
its benefits are not a math equation where two times the 
bits equals two times the performance. The biggest barrier 
to x64 adoption will be updating the skills and processes 
of your IT organization since switching means changing 
nearly everything, including ensuring all drivers and 
software have a 64-bit version. DBAs wanting to deploy a 
64-bit SQL Server 2008 implementation should work with 
the other departments in their IT organization to make 

it possible, if 32-bit is still the platform of choice at their 
company. In addition to updated skills and processes, you 
will also need to check the availability of the software 

and utilities for x64 such as anti-virus and monitoring as 
well as hardware drivers such as the ones for HBA cards. 
Check with your preferred vendors to see if they have x64 
versions for the software and hardware you deploy or will 
be deploying. The same rules will also apply in a transition 
to Windows Server 2008 — are your organization's skills 
and processes updated, and do you have equivalents for 
your tools for that platform? Windows Server 2008 should 
be seriously considered for SQL Server 2008 as it will help 
with increased performance and other aspects of SQL 
Server such as security. 


A big fallacy when it comes to servers — especially in 

the x64 age - is that you can get away with only blades 
or smaller 1U servers and achieve the same scale and 
performance. To some degree that is true, but there is 
still a need for larger, "big iron" servers depending on 
your requirements. All servers have a point at which they 
cannot scale. Will you hit it sooner or later on a blade 

or 1U? While there are no guarantees, if you purchase 
hardware with limited capacity for expansion, there will 
be a hard limit to how far you can go. If you're trying 

to run your largest, mission-critical system accessed by 
10,000 employees that needs to be up 24x7, be in place 
for five years, and be highly performing that entire time 
going large is going to offer you much more flexibility and 


growth than a small server. Part of the reason there is a 
drive towards consolidation and virtualization is that this 
SQL Server sprawl with small servers has inundated many 
IT organizations. 


Applications 


Applications hold the key to the performance of your 
databases. If a database server had no applications to 
connect to it, it would remain highly performing since it 
would have little demand on processor, disk, or memory. 
Database servers need to be sized appropriately. The 
problem is that most database servers are bought well 
before the application has even been developed. The 
process is usually something like this: 


e A decision is made to deploy an application 


e Due to the lead time for ordering and receiving servers, 
then racking and stacking them (at least a three month 
process), production servers are ordered with no real 
information to go on 


* The server goes into production and is often either 
too large or underpowered. Too large is fine from a 
performance standpoint however it may be seen as 
wasteful by IT since the server is not utilized "enough". 
Underpowered means that you are already starting 
behind the proverbial 8-ball from day one. The goal is to 
achieve a happy medium. 


One of two things needs to happen when you place 
future server orders. The best method of sizing servers 
properly is to develop an application, or get a third-party 
application in and test your workload against it. Even 

if you test on a smaller server to measure processor, 
memory, and disk usage, you can extrapolate numbers. 
Here are some items to collect and measure: 


* The overall processor utilization on average. 


* The amount of memory and processor used by the SQL 
Server process — not just the server alone. 


* Know how tempdb is being used. Some databases and 
applications use it heavily, others not. This will affect the 
disk configuration and ultimate performance of your 
SQL Server instance. 


* Know the ratio in your application of reads to writes. 
This will not only help determine growth, but how to 
help measure your I/O performance. 


e Physical disk utilization to measure the amounts/of I/O 
being consumed both in terms of size and the time it 
takes to complete a read or write operation (captured in 
counters such as the Average disk/sec read or write in 
Performance Monitor in Windows). The measurements 
of I/Os will open a dialog with the storage administrators 
and storage vendors to get you what you actually need 


for disk performance. The choice of your backup method 
will also determine the scale for the log file 


* Know what the system looks like at rest (generally 
measured before the system goes into production), at its 
busiest (for example, month end close), and what it looks 
like during normal load. This is especially important if 
you are re-deploying a SQL Server for an already existing 
application. You should have performance data going 
back to the initial day of production to measure growth. If 
you do not, start now. After deploying SQL Server 2008, 
you can use the new feature called Data Collector to start 
capturing and analyzing performance data. 


You do not need to measure every performance counter 
every second of the day on every server. Be smart - it will be 
virtually impossible to analyze that much data. Only capture 
what is needed and expand from there as needed. 


It should be pointed out that the biggest key to testing an 
application is to ensure that it is not only tested for features 
(generally done in singular tests), but under load. Without 
throwing your actual workload at a system, any sizing is a 
guess. 


rowth 
pie is directly related to usage. The most important 
thing to account for when sizing servers is how long you 
plan on having this entire solution in place. No one sets out 
to buy servers to only be in production for two months; the 
average lifespan for most servers tends to be three to five 
years or longer. That means a server purchased in 2008 may 
be in production until 2013. That is a long time. 


Going back to application testing, knowing the signature of 
that application from the memory and processor standpoint 
will help. For example, if you measure on a test system 

that you are already at 3596 CPU utilization with 4,000 
employees using the application but plan on expanding 

to 8,000 employees in 12 months, if the CPU load nearly 
doubles in that timeframe, you will be dangerously close to 
capacity with four years to go. 


When it comes to disk, as applications have become more 
and more complex, they are utilizing and storing increasing 
amounts-of data. This data is also becoming larger to store 
with the introduction of unstructured, BLOB, and geospatial 


data. Since disk is seemingly cheap, application owners are 
often assuming IT is going to be able to compensate for their 
storage demands used by a feature. This is a poor assumption. 
While disk is cheaper, it is not free. There is a finite amount 
that any storage can scale before needing to buy more disks 
or hardware. Storage administrators and DBAs need to be in 
conversations with application owners and developers long 
before it is deployed; at that point, it is too late. 


Another issue related to disk and growth is regulation by 
government or external entities. Many industries must keep 
data around longer, whether it is in the main database or 
keeping backups around for many years. This puts a drain on 
all IT resources including DBAs since the cost of maintaining 
that much data gets harder, not easier, as time goes on and 
windows for performing preventative maintenance such as 
index rebuilds will consume more time than an outage will 
allow. IT will suffer under the strain of retention if it is not 
accounted for up front. 


Finally, there is the simple math of disk growth: knowing how 
big the data and log files will grow in addition to backup 
storage and its requirements. Every application vendor or 
owner should be able to tell you the average size of an insert 
so DBAs can do calculations on size usage. For example, if an 
application uses 20KB per insert, and there are 1000 inserts 
an hour that equates to just under 20MB/hour. That equates 
to about 175GB/year in growth. If that is based on current 
application usage, not future (which will change over time), 
you need to figure out as per the example above how adding 
employees may change this number. Remember that the log 
file has to account for tasks such as index rebuilds, so as the 
data usage grows, so will log usage. 


When it comes to backups, backups generally grow 
proportionally to data size. Even if you are using SQL Server 
2008 Enterprise Edition's backup compression or one of the 
third-party vendor backup compression utilities, they will not 
solve all storage problems. They help, but if you need to store 
multiple backups on disk — especially with VLDBs — it will be 

a challenge. Be smart from the start, and you will stay ahead 
of, not behind, the problem. 


Avatlabilify 


Poor performance often leads to poor availability since 
queries that take longer, issue table scans (instead of hitting 
indexes), locks and blocks, and other issues can lead to a 


Without throwing your actual 
workload at a system, 
any sizing is a QUuess. 


downward spiral to the point of total unavailability of a 
database or application. If you need to take unplanned 
downtime to fix a performance issue, that's an outage 
that contributes to missing availability SLAs. One of 

the most common areas of reconfiguration is the disk 
subsystem. Perception is also reality: if end users think 
that the system is slow, it is often considered an outage 
since the application may be virtually unusable. Never let a 
performance issue become an availability problem; this can 
generally be avoided by proper maintenance and sizing. 


It is no longer a 9 to 5 world — servers and systems need to 
meet the demands of global businesses and customers at 
any hour of the day. That does not happen easily and takes 
reliable hardware along with careful planning, deployment, 
and ongoing, proactive administration. Availability is 
dependent on formal service level agreements (SLAs) set 
up between the application units or the business and IT. 
SLAs extend to performance as well. Formalizing availability 
and performance SLAs allows both sides to measure and 
be accountable. Without SLAs, it is impossible to size and 
deploy systems because there are no formal guidelines to 
measure against. The SLAs should be continually monitored 
and revised as needed. 


Consolidation 


Consolidation is driving a need for SQL Servers that are 
more highly performing than ever. Most companies have 
varying numbers of older, out-of-date, inefficient, under- or 
over-utilized servers that would benefit from consolidating 
to fewer servers and/or instances of SQL Server. Optimizing 
a SQL Server environment through consolidation can 
provide a measurable, long-term benefit not only to the IT 
organization, but to a business’ bottom line. The choices 
for consolidation are varied: single or multiple instances on 
a single server or cluster or virtualization with a technology 
such as Windows Server 2008 with Hyper-V. Your strategy 
will depend on your needs, but SQL Server supports both 
approaches. Many times, it may be worth considering one 
larger server that can either be partitioned into smaller 
servers or used as one very large, very powerful server that 
can host many virtual servers with Hyper-V. The.ácquisition 
cost may pay for itself over time, and it should be 
considered a viable option when considering consolidation 
of SQL Server databases and instances. 


Budgets 

The final factor for performance is not an obvious 

one: your budget. The architect mày have visions of 
the ultimate scalable system when designing, but as 
soon as he or she sees the amount that is allocated, 
reality smacks them in the face. Very few companies 
give their IT organization carte blanche when it comes 
to ordering systems. This is true whether or not you 
have hundreds of millions of dollars in the bank or just 
hundreds. Money is tighter everywhere, so ordering the 


right server for a reasonable 


price is going to make 


or break the implementation long term. Making a 
choice largely based on price is a recipe for disaster. 

This is often in direct conflict with most organizations. 

It is not uncommon for an IT shop to have three to 

five "standard" server builds to choose from, and an 
implementation has to fit into one of them. It is a bit like 


trying to put a square peg in 
other needs, you either cann 
jumping through hoops to g 
and other hardware compon 


a round hole. If you have 

ot get it or it takes a lot of 
et there. Deploying servers 
ents that are not adequate 


will cause availability and performance issues sooner 
rather than later, and the price to fix those problems is 
often more expensive than if the proper solution had 
been purchased from the start. Cost cannot be ignored, 
but spend wisely. Compound cost with preferred 
hardware vendors where you can only purchase from 
certain companies as well as be limited by hardware 
selection, and the drive to over-consolidate or virtualize 
things does not get easier. The right solution will be a 
compromise. 


Lontlusion 


Deploying highly performing SQL Server systems is not 
just a DBA or IT problem. Proper performance begins 
from day one of any solution or application’s planning 
and requirements — well before implementation or a line 
of code is written. Often times DBAs are not involved 

at this phase, but pay the price for it later. DBAs can 

help by assisting in the process to size, acquire, and 
deploy servers as well as ensure that the proper proactive 
maintenance is in place to keep the databases and 
instances running like well-oiled machines. The best 
thing you can do now in your current environments if 
you are not already is to start to define, analyze, monitor, 
and project what your various workloads are and will 

be. The more information you have up front, the better 
the conversations will go with both internal groups as 
well-as-with-hardware vendors. Then and only then can 
you implement and deploy appropriately priced and 
sized servers for SQL Server that meet the performance, 
availability, and growth needs for the business. 


Allan Hirt has been using SQL Server in various guises 
since 1992. For the past 10 years; he has been consult- 
ing, training, developing content, speaking at events, 
and authoring books, whitepapers, and articles. His most 
recent major publications include the book Pro-SQL Server 
2005 High Availability (Apress, 2007) and various articles 
for SQL Server Magazine. Before striking out on his own 
in 2007, he most recently worked for both Microsoft and 
Avanade, and still continues to work closely with Micro- 
soft on various projects. He can be reached via his Web 
site at http:/Awww.sqlha.com or at allan@sqlha.com. 


from PctRankRanges. This is achieved with 
the predicate: 


(@mark > mark from AND @mark <= mark to) 
OR (rownum = 1 AND @mark = mark from) 


A CASE expression in the SELECT list 
determines whether interpolation is required. 

If the input mark (@mark) is equal to one of 
the existing marks in the range, the CASE 
expression returns the corresponding per- 
centile rank; otherwise, the CASE expression 
uses the aforementioned formula to apply 
interpolation. The code in Listing 4 returns the 
value 0.766667 (76.6667 percent). 

With minor revisions to the code in List- 
ing 4 you can calculate the percentile ranks 
of multiple values stored in a table. Listing 5 
shows how to do so. The code populates a table 
variable called @MyMarks with a few random 
marks. The outer query in Listing 5 joins the 
table variable with PctRankRanges, to apply 
the calculation to each mark from the table 
variable as opposed to a single mark in a scalar 
variable. When I ran the code in Listing 5 on 
my system I got the output shown in Table 
3, page 26. Note that the calculation returns 
NULLs for marks that are smaller than the 
minimum or higher than the maximum in the 
Marks table. 


Percentiles 

As I explained, a percentile is a value below 
which a certain percent of values fall. PH 
describe the way Excel calculates the PERCEN- 
TILE function, then I'll explain how to imple- 
ment similar logic with T-SQL. 

Given an ordered set of values vl, v2, ..., 
vent, and a request for percentile pct, Excel’s 
PERCENTILE function calculates the pct 
percentile as follows: 


Calculate n as pct * (cnt - 1) + 1 


Using the marks from our Marks table as an 
example, the ordered set of marks is: vl = 40, v2 
= 55, v3 = 70, v4 = 70, v5 = 70, v6 = 75, v7 = 75, 
v8 = 80, v9 = 80, v10 = 95, v11 = 100. In our case 
cnt = 11. For pct = 0.95 (95th percentile) n = 0.95 
*(11-1)+1= 10.5. 

Now, let k be the integer part of n and d be 
the decimal part of n. In our case, k = 10 and d 
— 0.5. If d 2 0, return vk. That is, when n is whole 
(n = k), return the k^? value in the ordered set. If 
d <> 0, interpolate to produce the percentile as 
follows: 
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CALCULATE PERCENTILES 


TABLE 2: Output of Query in Listing 3 


rownum mark from mark to petrnk to 
NE pctrnk from M M 

1 40 55 0.00000000000 0.10000000000 
2 55 10 0.10000000000 0.20000000000 
3 10 15 0.20000000000 0.50000000000 
4 15 80 0.50000000000 0.70000000000 
5 80 95 0.70000000000 0.90000000000 
6 95 100 0.90000000000 1.00000000000 


When k = cnt, return v, , + d * (v, - v, )) 
When k < cnt, return v, + d * (v,,, - v) 


In our case, k < ent (10 « 11), therefore the cal- 
culation is v, + d * (v,,, - v,): 95 + 0.5 * (100 - 95) 
= 97.5. 

The code in Listing 6, page 26, implements 
this logic and calculates the percentiles for a 


LISTING 5: Query Calculating Percentile Ranks 
for Multiple Random Values 


k+1 


DECLARE @MyMarks TABLE(mark INT DEFAULT (ABS((CHECKSUM(NEWID())))%181)) ; 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 
INSERT INTO @MyMarks 


DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 
DEFAULT VALUES; 


WITH MarksRnkCnt AS 
C 


SELECT studentid, mark, 
RANK() OVER(ORDER BY mark) AS rnk, 
DENSE RANK() OVERCORDER BY mark) AS drnk, 
COUNT(*) OVERO AS cnt 
FROM dbo.Marks 
DE 
PctRanks AS 


SELECT DISTINCT 
drnk AS rownum, 
mark, 
1.*(rnk-1)/(cnt-1) AS pctrnk 
FROM MarksRnkCnt 
), 
PctRankRanges AS 
C 


SELECT 
Cur . rownum, 
Cur.mark AS mark_from, Nxt.mark AS mark_to, 
Cur.pctrnk AS pctrnk from, Nxt.pctrnk AS pctrnk to 
FROM PctRanks AS Cur 
JOIN PctRanks AS Nxt 
ON Nxt.rownum = Cur.rownum + 1 
) 
SELECT M.mark, 
CASE M.mark 
WHEN mark from THEN pctrnk from 
WHEN mark to THEN pctrnk to 
ELSE 
pctrnk from 
+ 1.*(M.mark - mark from) 
/ (mark to - mark from) 
* (pctrnk to - pctrnk from) 
END AS pctrnk 
FROM GMyMarks AS M 
LEFT OUTER JOIN PctRankRanges AS P 
ON (M.mark » P.mark from AND M.mark «- P.mark to) 
OR (P.rownum = 1 AND M.mark = P.mark from) 
ORDER BY M.mark; 
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TABLE 4: Output of 


TABLE 3: Output of 
Query in Listing 5 


CALCULATE PERCENTILES 


whole set of percents 
represented by the CTE 
called Pets. Play with 


mark petrnk 

your own percents to 
aes aM] vtr rusos see how the output 
3 NULL changes. The CTE 
16 NULL called CntMarks has 
16 NULL a single row with the 

value cnt representing 
2 Me the count of rows in the 
61 0.140000 Marks table. The query 
62 0.146667 defining the CTE called 
71 0.260000 PctCnt_n calculates for 

each percent from Pcts 
El SES the value n described 
2 Udi) earlier. The query in the 
91 0.846667 CTEcalled PetCnt_ndk 


breaks n to its integer 
and decimal components k and d, respectively. 
The query defining the CTE called MarksRn 
simply calculates row numbers (column rn) for 
the Marks from the Marks table. 
Think of the row numbers as the 
positions in the ordered set (1, 2, ..., 


mark 70, and so on. 

Note that Excel interpolates to calculate the 
percentile when n is not whole. Other ways to 
calculate percentiles are to round n and return 
the value in the rounded position in the ordered 
set, but PII leave this method for you as an 
exercise. [Editor’s Note: If you try this method, 
please email us with your experiences. Send your 


feedback to Ipeters(gpenton.com.] 


Statistically Speaking 
You can easily use T-SQL to implement statis- 
tical calculations such as percentile and percen- 
tile rank. In addition, you'll likely encounter 
many other Excel-supported statistical calcula- 
tions and other types of calculations that are 
handy for use in a database. Besides the useful- 
ness of such functions in statistical analysis, 
trying to implement them with T-SQL 1s a great 
exercise. [SQL 
InstantDoc ID 100129 


LISTING 6: Query Calculating Percentiles 


Query in Listing 6 


pet 

0.05 
0.10 
0.15 
0.20 
0.25 
0.30 
0.35 
0.40 
0.45 
0.50 
0.55 
0.60 
0.65 
0.70 
0.75 
0.80 
0.85 
0.90 
0.95 
1.00 
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. WITH Pcts AS 
cnt), and the corresponding marks œ 
SELECT 9.05 AS pct 
as vl, v2, ..., nM eq at UNTONLAET- SELECT ate 
oins PctCnt n an UNION ALL SELECT 0.15 
een matching UNION ALL SELECT 9.28 
5 UNION ALL SELECT 9.25 
to each percent from ron ALL SELECT 8.35 
5 rel- UNION ALL SELECT 9.49 
percentile PetCnt_ndk the rel UNION ALL SELECT 8.45 
evant marks from UNION ALL SELECT 0.56 
See ep a : UNION ALL SELECT 9.55 
47.50 MarksRn. To remind UNION ALL SELECT 9.66 
: = UNION ALL SELECT 9.65 
you, when d 0, the UNION ALL SELECT 9.78 
2900 relevant mark is v; Map 
62.50 else, when k = cnt, the UNION ALL SELECT 9.85 
E UNION ALL SELECT 0.90 
70.00 tworelevant marks are Te seer dS 
Va and Vo else, when UNION ALL SELECT 1.00 
70.00 i yp 
k < cnt, the relevant — Cntmarks As 
70.00 marks are v, and v, 
k k+l" SELECT COUNT(*) AS cnt FROM dbo.Marks 
70.00 In all three cases, for [Peo As 
70.00 each percent, the for- < 
s E SELECT pct, cnt, pct * (cnt - 1) -« 1 AS n 
7250 mulas for calculating FROM Pcts 
mm the percentiles can be  }, 6309S J0IN CntMarks 
generalized to: cena AS 
75.00 SELECT pct, cnt, n, CAST(n AS INT) AS k, n - CAST(n AS INT) AS d 
FROM P: 
75.00 MIN(mark) + d Sa 
77.50 * (MAX(mark) MarksRn AS 
- MIN(mark)) SELECT 
80.00 ROW NUMBER() OVER(ORDER BY mark) AS rn, 
mark 
80.00 The output of the FROM dbo.Marks 
) 
80.00 i isti i SELECT pct, 
code m Listing 6 I$ MIN(mark) + d * (MAX(mark) - MIN(mark)) AS percentile 
87.50 shown in Table 4. As FROM PctCnt ndk AS P 
: h JOIN MarksRn AS M 
95.00 you can see in the ON (d = 0.00 AND rn = kK) 
: OR (d > 0.00 AND k = cnt AND rn IN(k, k-1)) 
97.50 t d a OR (d > 8.08 AND k < cnt AND rn INCk, kk) 
is the mar , the GROUP BY pct, k, d 
100.00 ORDER BY pct; 


lower quartile is the 
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Did You Know... 


Along with windowsitpro.com and sqlmag.com two new sites 
have been launched to ensure custom-made content is just a click away. 


Office & a 
na PRO windowsdev pro.com 
Microsoft Office and SharePoint content mentored by A community addressing the need of content for the 
a community of peers and professionals. developer who needs to create with the IT administrator in mind. 
Www.officesharepointpro.com www.windowsdevpro.com 


Windows t! pr 


Engage with our network of peers and professionals and view various forms of content. 
Itis a complete source for IT Professionals and managers. 
www. windowsitpro.com 


EmiServer 


or information on managing, mining, building and developing world-class applications. 
WWW.sqlmag.com 


Data Dynamics 


z 


Az Analysis 
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WHEN DATA DYNAMICS PROVIDES THE TOOLS TO ADD BI TO 
YOUR OWN APPLICATIONS, WHY SETTLE FOR GENERIC? 


INDUSTRY LEADER FOR OVER 

12 YEARS 

Data Dynamics has been an innovator 
and trusted leader in the BI market 
since 1996. 


OVER 60% OFF THROUGH NOVEMBER SOTH! 


Data Dynamics 


YOUR DATA IS A HIDDEN TREASURE, DIVE IN 


GAIN DEEP INSIGHT TO DATA 
Discover hidden gems in your data that might otherwise go unnoticed. 


MODEL UNSTRUCTURED DATA 
Transform relational data into logical multi-dimensional objects using a 
simple configuration file. 


CUSTOMIZED SECURITY 
Provide role based security to deliver detailed level access to any number 
of users. 


lg ‘ Data Dynamics 


«^ Business Intelligence 


FREE SUPPORT 

All Data Dynamics products have 
free support through email, forum or 
phone for the life of the product. 
SUPPORTS ANY DATABASE 
Don't settle for tools that only support | NO MAINTENANCE FEES 
certain data sources, our components | Free updates improve 
support anything you can read into | applications over time. 

your own application. 


your 


ROYALTY FREE DISTRIBUTION 
Distribute your application, no matter 
how big your enterprise is, without 
paying another dime. 


GO AHEAD, SHOW OFF 


WORD/EXCEL/PDF EXPORTS 
Take reports with you in any of the supported formats to: send out, print 
out, or just analyze the numbers. 


E 


CALENDAR 
View and analyze date based data. 


dashboards with the ne 


Integrate 


Use SSIS to retrieve data from a MySQL 
database and insert it into SOL Server tables 


you work with SQL Server Integra- 
S tion Services (SSIS), sooner or later 
you'll have to retrieve data from 


a MySQL database. Fortunately, MySQL provides 
drivers that make the process relatively easy. And once 
you learn the basics of how to use those drivers, you'll 
be able to integrate MySQL data almost as easily as 
SQL Server data. To help you get familiar with using 
SSIS with MySQL, I'll show you an SSIS solution— 
MySQL SQLServer—that retrieves data from two 
tables in a MySQL database and inserts that data 
into comparable tables in a SQL Server database. In 
our discussion here, I assume that you already know 
how to develop an SSIS package in Business Intel- 
ligence Development Studio, the pared-down version 
of Microsoft Visual Studio 2008 that ships with SQL 
Server 2005. I also assume that you're familiar with 
how to define connection managers, control flow, 
data-flow elements, and Windows ODBC data sources. 
You can find information about MySQL at www.mysql 
.com. To download the generally available edition of 
MySQL, go to the Developer Zone. 


Step I: Connecting to MySQL 

The first step in creating an SSIS solution that connects 
to a MySQL database is to install the driver needed to 
connect to that database. MySQL provides a number 
of drivers (called connectors) that let you connect to a 
MySQL database. To connect from SSIS, you can use 
the Connector/Net driver or the Connector/ODBC 
driver. The Connector/Net driver provides an ADO 
.NET interface to the MySQL database, and, as the 
name suggests, the Connector/ODBC driver provides 
an ODBC interface. In the MySQL. SQLServer solu- 
tion, I demonstrate both connectors. 

You can download the two connectors at www. 
mysgl.com/products/connector. The site provides 
information about downloading and installing the 
connectors and points you to other references that 
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100114 (click the Download the Code link at 


explain how to use the connectors. I encourage you 
to review these materials before installing either con- 
nector. Because the site so thoroughly documents each 
driver, I won't go into the driver details here. However, 
to create the MySQL. SQLServer solution, you'll need 
to download and install both connectors. Follow the 
instructions on the MySQL website, and for both con- 
nectors, perform a default installation. 

To use the Connector/ODBC driver, you must 
create a Windows ODBC data source after you've 
installed the driver. The data source should point to a 
MySQL database. For this article, I’m using the Book- 
store database, which contains two tables, Authors and 
Books. I wrote an SQL script, MySQL, BookstoreDB, 
to create the database and populate the tables with a 
few initial rows; you can download this code, as well 
as the other files for the MySQL SQLServer 
solution, at www.sglmag.com, InstantDoc ID 


the top of the article). You'll need to run this 
script against a MySQL installation before 
you can implement the solution. Note that, for this 
solution, I installed MySQL 5.0 (a default installation) 
and SQL Server 2005 Developer Edition on the same 
Windows XP computer. 

After you create the Bookstore database in MySQL 
and install the Connector/ODBC driver, you can create 
the Windows ODBC data source. To support my solu- 
tion, I created a system Data Source Name (DSN) 
ODBC data source called mysql and selected MySQL 
ODBC version Driver (aka the Connector/ODBC 
driver; the driver is displayed with this name after it's 
installed) from a list of existing drivers. Note that if 
you installed MySQL on a port other than the default, 
you'll need to enter the correct port on the Connect 
Options tab in the Connector/ODBC Configure 
Data Source Name dialog box. For more information 
about creating an ODBC data source, see Windows 
Help. (You don't have to perform any steps to use the 


FEATURE. 


Robert Sheldon 
contact@ rhsheldon.com) is a technical con- 


sultant and author of material about Windows, 
relational database management systems, and 
business intelligence design and implementation. 
His latest book is Beginning MySQL (Wiley). 
Find out more at www.rhsheldon.com. 
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Download the code at 
InstantDoc ID 100114. 
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EK. Advanced E ditor for Datafieader Source - OLEDB 


W-D DataReader Eror Output 


Figure | 


Reviewing the BooklD 
column’s data-type 
properties 


E. SOL Destination E ditor 


20 

output column "BookiD" (20) 
20 

BookiD 


0 
0 
lon AD. 


0 


fourbyte unsigned integer [DT_Ut4] 
0 


0 


Connector/Net driver; it’s ready to go as soon as you 
install it.) 


Step 2: Defining Connection 
Managers and Control Flow 
Now that you've set up your drivers and ODBC 
data source, you can create your SSIS solution. You 
can create the solution in either Business Intelligence 


Figure 2 


Mapping columns in a 
SQL Server destination 
component 
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Development Studio or the full version of Visual Studio 
2005 or later. (Use the Integration Services Project 
template to create your SSIS package.) When creating 


the SSIS package, you should start by defining 
the connection managers. But before you do 
so, run the SQLServer_BookstoreDB script, 
which you can also download at InstantDoc 
ID 100114, to create the target database (Book- 
store) in SQL Server. Then create a connection 
manager to the Bookstore database. 

To test both MySQL drivers, you need 
to create a connection manager for each 
one. Let’s start by creating the connection 
manager for the Connector/ODBC driver or 
more specifically, for the mysql DSN. Intui- 
tively, you d think that you simply create an 
ODBC connection manager. However, this 
approach doesn’t work. Instead, you must use 
a Microsoft .NET Framework ODBC data 
provider, which is an ADO.NET connection 
type, to create the connection manager. To 
create an ADO.NET connection, right-click 
the Connection Managers pane in Business 
Intelligence Development Studio, then click 
New ADO.NET connection. In the Configure 
ADO.NET Connection Manager dialog box, click 
New. In the Connection Manager dialog box, select 
.Net Providers\Odbc Data Provider from the Provider 
drop-down list. Also select mysql as the system DSN. 
Because the mysql DSN includes the username and 
password, you don’t need to specify it here. 

Now create the connection manager for the 
Connector/Net driver. Once again, you create an ADO 

.NET connection. However, this time you should use 
the provider .Net Providers MySQL Data Provider. In 
the Connection Manager dialog box for this provider, 
you must enter the name of the MySQL database, 
server, user, and password. In our example, I specified 
Bookstore as the database and localhost as the server. In 
addition, I used the root user account to access MySQL, 
but typically you'd use a more restrictive account. Note 
that after you enter the password and close the dialog 
box, the password is no longer displayed. 

The next step in creating the SSIS solution is to 
add two Data Flow tasks to the control flow, one for 
each MySQL driver. Each task will retrieve data from 
one of the tables in the MySQL database. For this 
solution, I used the Connector/Net driver to retrieve 
data from the Books table and the Connector/ODBC 
driver to retrieve data from the Authors table. After 
you add the Data Flow tasks, you can view them 
along with the connection managers you configured 
(MySQL Net and MySQL. ODBC, in our example) 
on the Control Flow tab in SSIS Designer. 


Step 3: Configuring the Books 
Data Flow 
After you add the Data Flow tasks to the control flow, 
you can begin to configure the Books data flow. Start 
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by adding a DataReader source component. To 
do so, open the component’s editor. Then, on 
the editor’s Connection Managers tab, specify 
the connection manager for the Connector/Net 
driver. In my solution, I named the connector 
MySQL_Net. 

Next, on the Component Properties tab, add a 
SELECT statement to the SqlCommand property 
to retrieve the data from the Books table. I used 
the statement 


select * from Books; 


I left all other properties with their default values, 
as I did on the Column Mappings and Input and 
Output Properties tabs. However, there are a few 
items on the Input and Output Properties tab that 
are important to note. 

If you expand the Input and Output Properties 
tab's DataReader Output tree, as Figure 1 shows, 
you'll see a list of input (external) and output 
columns. Look at the BookID output column. As 
you can see, the data type assigned to this column 
is a four-byte unsigned integer, which is consistent 
with the column created in MySQL. (An unsigned 
integer is one that doesn’t support negative numbers.) 
However, the BookID column in SQL Server wont 
accept the value in this format, which means that you'll 
need to convert the column to the correct data type. 
You'll run into the same problem with the NumIn- 
Stock column, although this is a two-byte unsigned 
integer, rather than four-byte. 

Now look at the Title output column’s properties. 
Notice that the data type is Unicode string. Although 
the MySQL column definition doesn’t specify Unicode, 
this is how the data is saved. However, this isn’t how 
the column was created in SQL Server. Consequently, 
you'll need to convert this column, too. You don’t have 
to worry about the DateModified column’s data type. 
SSIS retrieves this as a timestamp column, which is 
compatible with SQL Server’s DATETIME column. 

Because you have columns that are incompatible 
between the source and destination, you'll have to 
convert those columns. To do so, your next step is to 
add a Data Conversion transformation to the data flow 
and connect the source to the transformation. 

Now open the transformation's editor. You'll 
need to convert the BookID, Title, and NumInStock 
columns, as Figure 2 shows. As you can see, you must 
convert the unsigned integers to signed integers of the 
same size and convert the Unicode string to a regular 
string. When you convert the string, leave the length as 
60. This 1s the column length in both the source and 
destination databases. Also assign new column names 
to the outputted data. I simply added the number 2 to 
the names. 
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Finally, after you've converted the data, you can 
send it to SQL Server. To do so, add a SQL Server 
destination component to the data flow and connect 
the transformation to the destination. In the compo- 
nent's editor, select the connection manager for the 
SQL Server database and the Books table from that 


E. Advanced E ditor for DataRieader Source - ODBC 


database. For this solution, I’ve named the connection 
manager BookstoreDB. 

Now make sure that you correctly map the columns 
from the transformation to the destination. Look at 
the Mappings page of the destination's editor, and 
you'll see that I mapped the converted columns to 


Figure 3 


Books data flow 


426 
output column "Author D" (426) 


Figure 4 


Reviewing the 
AuthorlD column's 
data-type properties 
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§ B Derived Column Transformation Editor 


Replace TName | TRIMIFNane] 


| Unicode string [DT WSTR] | a 


Replace MName' | TRIMIMName] | Unicode sting [OT_WSTR] | 21 


Replace 'LName* 


TRIM[LName) ^ | Unicode sting [DT WSTR] | 21 


Figure 5 


Configuring a Derived 
Column transformation 


Figure 6 


Authors data flow 
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their respective destination columns but left the Date 
Modified column with its default mapping. 

That's all there is to it. As with any SSIS solution, 
the key is to make certain that the source data is prop- 
erly converted to the format necessary to be inserted 
into the destination database. You can see the final 
Books data flow in Figure 3, page 31. Now let's look 
at the Authors data flow. 


Step 4: Configuring the 
Authors Data Flow 

As you did for the Books data flow, your first step is 
to add a DataReader source component, configure 
it with the correct connection manager (in this case, 
MySQL ODBC), and define the SQL statement 
needed to retrieve data: 


select * from Authors; 


Now look at the Input and Output Properties tab. 
This is where things get interesting when using the 
Connector/ODBC driver. You can see the AuthorID 
column properties in Figure 4, page 31. Notice that, 
unlike the BookID column, the data type is now an 
eight-byte signed integer, although the AuthorID 
definition is the same as BookID. Thus, you'll need to 
convert AuthorID to a four-byte signed integer data 
type. Now look at the FName column's properties. 
As with the Title column, the data type is a Unicode 
string; however, notice that the length is 21, although 
the column definition in MySQL specifies 20. The 
MName and LName columns exhibit the same 
behavior. 

When you find a discrepancy of this nature, keep an 
eye out for the driver adding an extra space before or 
after each value. To protect against this possibility, you 
can use a Derived Column transformation to trim the 
extra spaces. Figure 5 shows how youd trim the three 
columns that contain a string value. Notice that you 
use a TRIM function and simply replace the column 
with the trimmed values. 

After you create your derived columns for the 
string values, you can then add a Data Conversion 
transformation and connect the Derived Column 
transformation to the Data Conversion transforma- 
tion. In the Data Conversion transformation, convert 
the AuthorID column to a four-byte signed integer and 
convert the three name columns to a regular string with 
alength of 20. You don't have to worry about the Date- 
Born or DateDied columns. Although they're defined 
as DATE columns in MySQL, SSIS imports them as 
timestamp columns. From there, add a SQL Server 
source component and be sure to map the converted 
columns to their respective destination columns. When 
you're finished, open the Data Flow tab to view the 
Authors data flow that you've configured, as Figure 6 
shows. 


On Your Way to 
MySQL Integration 
The solution I’ve shown you is, of course, a basic one, 
but it does give you an overview of how to retrieve 
data from a MySQL database and some of the issues 
you might have to address when you retrieve that data. 
Although I showed you how to use both MySQL 
drivers, you'll no doubt use only one in your SSIS 
package, and for the most part, you'll probably want 
to go with Connector/Net. However, I wanted to show 
you both connectors in case you encounter situations 
in which Connector/ODBC has already been imple- 
mented or you have another reason to use ODBC. 
Regardless of the solution you ultimately implement, 
you should now have a good idea of how to access a 
MySQL database from SSIS. [SQL| 
InstantDoc ID 100114 
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Finding Your Top 10 
SQL Server Queries 


Take a little effort to identify your 
worst-performing queries 


b di already capturing SQL trace information 
through the use of server-side traces, which send 
results directly to files on a locally attached disk on the 
server. Server-side traces not only minimize server load 
during the tracing process but also give you an easy way 
to parse and analyze the information later. By copying 
the files to another machine for processing, you can 
negate any impact on the production server while you 
analyze the data. 

But what if you need to parse the trace informa- 
tion in a way that will let you aggregate the data to 
find your worst-performing and most-called queries, 
stored procedures, and so on? One simple solution is 


to use a T-SQL user-defined function called SQL - 
Signature (popularized by Microsofts Tom 
Davidson). There are more efficient but slightly 
more complicated methods for tackling this 
problem—I'll follow upin the SQL Server Magazine Per- 
formance Tuning and Optimization forum (sglforums 


.windowsitpro.com/web/forum/) with a few—but this 


solution should get you started. 


Replacing Information 

In “Generating Server-Side Traces” (see the Learning 
Path on page 35), I established that you can use the fn_ 
trace_gettable() function to read events and columns 


LISTING I: The SQL Signature Function 


CREATE FUNCTION dbo.SQL Signature (@p1 NVARCHAR(MAX), @ParseLength INT = 4000) 


RETURNS NVARCHAR (4000) 
AS 
BEGIN 


DECLARE @pos AS INT, @mode AS CHAR(10), @maxlength AS INT, @p2len AS INT 


DECLARE @p2 AS 


SET @maxlength = 


NCHAR (4000), @currchar AS CHAR(1), @nextchar AS CHAR(1) 


LEN(RTRIMCSUBSTRING(@p1, 1, 4000) ) ) ; 


SET @maxlength = CASE WHEN @maxlength > @parselength 
THEN @parselength ELSE @maxlength END 


Ser Gaos = ils SEP (32 = "s SET Clm 


WHILE (@pos <= @maxlength) BEGIN 


SET @currchar = SUBSTRING(@p1, @pos, 1) 
SET @nextchar = SUBSTRING(@p1, @pos+1, 1) 


IF @mode = ‘command’ BEGIN 


SET @p2 = LEFT(@p2,@p21len) + @currchar 


SET @p2len = @p2len + 1 


ge SET Ganre = °° s 
SET @nextchar = ‘’; SET @mode = ‘command’ ; 


ie (reulrrtelnele 10 (QUI). Cg? Oat gS? L9 VU JAN) 
@nextchar BETWEEN ‘Ø’ AND ‘9’ BEGIN 
SET @mode = ‘number’ 


SET @p2 = LEFT(@p2,@p2len) + ‘#’ 
SET @p2len = @p2len + 1 


END 
LE Gcurnchatr— 4: BEGIN 
SET @mode = ‘literal’ 


SET @p2 = LEFT(@p2,@p2len) + ‘#’’’ 
SET @p2len = @p2len + 2 


END 
END 
ELSE IF @mode = ‘number’ AND @nextchar IN (‘,’,’)’,’ 
8 Dep Sun veo BAS 
SET @mode= ‘command’ 
ELSE IF @mode = ‘literal’ AND @currchar = ‘’’’ 
SET @mode= ‘command’ 
SET @pos = @pos + 1 
END 
RETURN @p2 
END 
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» 
à 
erred 


If you tackle 
the top 10 
offenders 
in your 
system, 
you'll 
probably 
tackle 90 
percent of 
the overall 
problems. 


TOP 10 QUERIES 


directly from the trace files. This function will be your 
primary means of retrieving trace data for direct pro- 
cessing (or placing it in a table for later processing). 


SELECT * FROM ::FN TRACE GETTABLE('C:N 
YourTraceFile.trc', DEFAULT) 


The goal of the SQL Signature function, which 
Listing 1, page 33, shows, is to replace certain pieces 
of information (which might vary between calls to the 
database) with a constant—in this case, the # symbol. 
These pieces of information are typically the values in 
a search argument of the WHERE clause that vary 
from one call to the next, while the rest of the statement 
remains constant. 

Replacing these values will let you get a core list of 
what I call Query Classes and will let you aggregate 
the data to get meaningful metrics. In the following 
example, the three statements are effectively the same 
except for the CustomerID value. If you tried to aggre- 
gate these with a simple GROUP BY clause, you would 
get three separate groups. 


SELECT * FROM [dbo].[OrderHeader] WHERE 
CustomerID - 397 


UR IT. TOOLBOX WITH ALL THE TOOLS YOU NEED 


PPS 


SELECT * FROM [dbo].[OrderHeader] WHERE 
CustomerID - 398 

SELECT * FROM [dbo].[OrderHeader] WHERE 
CustomerID - 398 


However, if you replace the IDs with a constant, you can 
group these together to get more meaningful metrics. 


SELECT * FROM [dbo].[OrderHeader] WHERE 
CustomerID - £ 


The same is true for stored procedure calls. For 
example, 


EXECUTE [dbo].[cp test] 79 
EXECUTE [dbo].[cp test] 71 
EXECUTE [dbo].[cp test] 72 
becomes 


EXECUTE [dbo].[cp test] # 


To use the function, you'd simply pass in the text 
you want to parse, along with a second parameter to 
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tell it how many characters from the beginning of the 
statement to use during the parsing. 


SELECT dbo.SQL Signature('select * from 
orders where orderid = 10255 AND MyDate 
= ''20060101''',1000) 


You can also use the function directly when reading the 
data from the trace file; to do it properly for many rows, 
you need to place the data into a table. You can use the 
following code to create the table, insert the data, and 
parse the TextData column: 


SELECT [EventClass], [TextData], [DatabaseID], 

[Duration], [Reads], [Writes], [CPU], 

dbo.SQL Signature([TextData],1000) AS 
[SQLSignature], 

CAST(@ as INT) AS [HashCode] 

INTO YourTable 

FROM ::FN TRACE GETTABLE('C: V YourTraceFile. 
trc', DEFAULT) 


Next, to make the aggregations easier, compute a 
hashed value of the parsed text for each row. You 
might want to add an index on HashCode after the 
update: 


UPDATE YourTable WITH (TABLOCK) 
SET [HashCode] = 
CHECKSUM( [SQLSignature]) 


Now, you can aggregate the data: 


SELECT [EventClass],SUMC[Duration]) AS 
[Total Duration], 
SUM([Reads]) AS [Total Reads], 
SUMC[Writes]) AS [Total Writes], 
SUMC[CPU]) AS [Total CPU], COUNT(*) AS 
[Total Counts], 

[HashCode], CAST('' AS NVARCHAR (4899) ) 
AS [SQLSignature], 

CAST('' AS NVARCHAR(4000)) AS [Sample - 
Statement] 

INTO AggTable 

FROM YourTable GROUP BY 
[EventClass], [HashCode] 


Finally, add the query class (SQLSignature) and 
a sample statement for each class, and you're 
ready to go. 


UPDATE AggTable SET [SQLSignature] - 

(SELECT TOP 1 a.[SQLSignature] FROM 
YourTable AS a WHERE a.[HashCode] = 

AggTable.[HashCode]), 

[Sample Statement] - 

(SELECT TOP 1 a.[TextData] FROM 
YourTable AS a WHERE a. [HashCode] 
AggTable. [HashCode]) 
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TOP 10 QUERIES 


Your Call 
You can easily add 
further metrics (e.g., 
averages, max) to this 
simplified example as 
you see fit. Now, to 
find your top worst- 
performing queries, 
you can select the 
aggregated results in 
any order you want— 
by Reads, Writes, CPU, 
or Duration. Also, 
take into account the 
number of times the 
query class was run. 
For example, if you execute a query once a day, and it 
has 1,000,000 reads, is that considered one of the top 
10 queries you should be concerned with? It isn't if you 
have other queries that use 5,000 reads each but are 
called once per second. Slice and dice it as you want, 
but if you tackle the top 10 offenders in your system, 
you'll probably tackle 90 percent of the overall prob- 
lems and get the biggest bang for your effort. SQL 
InstantDoc ID 100121 
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intelligence 
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help from SQL Server Magazine. 


data-delivery with front-end 
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EFFICIENT DATA BINDING with 
Windows Presentation 


Foundation 


and Visual Studio 2008 


Automate the development process 


2008 and found no support for an important type 

of data binding—automatically wiring up a data 
source (such as SQL Server) and a Windows Presenta- 
tion Foundation (WPF) window or page. I found the 
lack of support for this automation strange, especially 
considering that WPF is used to build rich UIs for 
Windows Vista applications. 


D my surprise when I opened Visual Studio 


My Tool to Automate 

Data Binding 

So, I decided to write my own tool to automate data 
binding with Visual Studio 2008 and WPF. I wanted 
to create a tool that's easy to use and generates main- 
tainable and reliable code. I was looking for three 
quality improvements: Better efficiency for my team 
because we can perform automated tasks much faster. 
Better adherence to standards because everyone uses 
the same tools to generate code. Solid code that passes 
testing 95 percent of the time or better. 

When I automate a process, I first figure out the 
pattern of the code I want to output. For instance, with 
WPF databinding you might find a code pattern that 
looks like this: 


With ComboBoxl 
.IsSynchronizedWithCurrentlItem = True 
.DisplayMemberPath = "CompanyName" 
.SelectedValuePath = "CompanyName" 
.ItemsSource = thisTableToBindDataDataView 
End With 


You should test several different patterns manually to 
determine which ones are the most effective. You want 
patterns that run well and are easy to maintain. If you 
don’t have a solid code pattern, then your tool will just 
write bad code faster. When you look at the code output 
of my automation tool, you'll see that the tool does 
all the data binding in .NET code, not in Extensible 
Application Markup Language (XAML). The reason 


SQL Server Magazine * www.sqlmag.com 


with a free new tool 


I chose this approach is simple—separation of code 
and UI. Use of the separation of code and UI code 
pattern is a basic component design principle. Plus, the 
XAML for the form might become really complex if 
all the data binding is there, so I didn't want to add to 
the complexity by mixing data binding code with it. For 
more information about automating processes, see the 
sidebar "Automating Development Can Lead to Project 
Success," InstantDoc ID 100363. 

To download the code for the automation tool go 
to www.sqlmag.com, enter 100362 in the InstantDoc 
ID text box, and click the 100362.zip hotlink. Open 
the WPFAutomaticDataBindingUI.sIn solution in 
Visual Studio 2008 and compile it. Then click F5 to 
open a project to test the add-in. To run the tool, open 
the WPF form you want to perform data binding on 
and run the tool by selecting WPFAutomaticData 
BindingUI from the Tools menu in Visual Studio. 

Figure 1, page 38, shows the UI after I ran the tool 
with a form open and then selected a property class 
(CustomersPropertyClass) to use as the data 
source for the form. The grid lists all the bind- 
able controls found on the form. Each drop- 
down list to the right of a control contains the 
names of the properties from the properties 
class. To automatically wire up a control to a property, 
simply select the property to bind the control to. When 
you finish selecting properties, click Finish to close the 
form and write the code into the WPF forms code 
behind file. 

TIl highlight some key elements of the code in the tool 
for you here, but I encourage you to download and look 
through all the code. For more information about WPF 
data binding and Visual Studio 2008 add-in creation, see 
the Learning Path for this article online at InstantDoc ID 
100362. 


Building the Tool 
Wiring up a Ul is simple, after you've figured out the 
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Contact Name TextBox 


Figure | 


Data binding tool with 
a simple form selected 


LISTING |: Code for 


Form Load 


Dim thisCustomerBL As New 


CustomersBusinessLayerClass 


Dim dt As New DataTable 


dt = thisCustomerBL.SelectAT] 


DataContext = dt 


SetupDataBinding(dt) 


EFFICIENT DATA BINDING 


data source to which you want to bind the UI controls. 
For example, you might bind to a web service method 
result, a message from Windows Communication 
Foundation (WCF), a proxy class, or a SQL Server 
database. You have many options to choose from when 
deciding what to point your UI to. For the purposes 
of this article, l'Il have the tool bind to two sources of 
data: Property Class and DataTable. 

Given these two sources, I can start to put together 
the code I need to retrieve the data. Listing 1 shows 
the data binding code I want to output when binding 
to a DataTable. The code calls the SelectAll method 
to return a data table of customers and then it sets the 
DataContext for the WPF window to that data table. 
The last line calls the SetupDataBinding function to 
perform the data binding magic. 

The real work in setting up data binding comes 
when you must wire up the property of a control to 
a field in a data source that contains the data you 
want to display. For instance, the SetupDataBinding 
code shown in Listing 2 calls the SetBinding 
method of the TextBox controls to perform 
the binding operation, while the ComboBox] 
has a slightly different syntax to do the same 
thing, as you can see in Listing 2. 

When I figured out how the WPF data 
binding syntax worked, it was easy to auto- 
mate. I chose not to output the form load 
code that gets the data from somewhere 


LISTING 2: Code for SetupDataBinding 


Sub SetupDataBinding(ByVal thisTableToBindDataTable As DataTable) 
Dim thisTableToBindDataDataView As DataView 
thisTableToBindDataDataView = thisTableToBindDataTable.DefaultView 
TextBox1.SetBinding(TextBox.TextProperty, "CompanyName") 
CityTextBox.SetBinding(TextBox.TextProperty, "City") 
AddressTextBox.SetBinding(TextBox.TextProperty, "Address") 
ContactNameTextBox.SetBinding(TextBox.TextProperty, "ContactName") 


With ComboBox1 


.IsSynchronizedWithCurrentItem = True 


.DisplayMembe 


rPath = "CompanyName" 


.SelectedValuePath = "CompanyName" 
.ItemsSource = thisTableToBindDataDataView 


End With 
End Sub 
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because that's usually harder to do, but it's 
simple code to write. Instead, I chose to 
automate the code output shown in Listing 
2 because that varies greatly from form to 
form and also can be quite complex on a 
form with a lot of controls. 


Creating the Tool 

Now, I'll show you how to generate the data- 
binding code using my automation tool. 
Figure 2, page 40, shows the two projects 
that make up the tool as seen in Visual 
Studio 2008 Server Explorer. The WPFAu- 
tomaticDataBindingUI project is just the 
add-in shell and references the WPFAutomaticData 
BindingLibrary, which contains the form (DataBinding- 
MainForm), shown in Figure 1, and the code. If you want 
to use this tool outside of Visual Studio 2008, you can 
modify the code in WPFAutomaticDataBindingLibrary 
by pulling out the references to Visual Studio 2008 and 
having the library read the WPF code from disk instead 
of from Visual Studio 2008. When the DataBinding- 
MainForm (in the WPFAutomaticDataBindingLibrary 
project) loads, it calls the LoadControlsToGrid method 
which in turn calls GetControlDefinitions to load the 
controls from the WPF definition (XAML code). Since 
XAML is XML, it’s easy to parse. The GetControl 
Definitions function grabs the XMAL from Visual 
Studio 2008 with the following lines of code: 


applicationObject.ExecuteCommand("View. 
ViewDesigner") 

txt = CType(applicationObject. 
ActiveDocument.Selection, EnvDTE. 
TextSelection) 

txt. SelectAl1Q 

xWPFElement = System.Xml.Ling.XElement. 
Parse(txt.Text) 


This code makes sure the window is in ViewDesigner 
mode then takes the XAML and puts it in the txt vari- 
able, which is of type EnvDTE. TextSelection. Finally, it 
loads the xWPF Element (System.Xml.Ling.XElement) 
with the XAML. You can work with it using LINQ and 
anything else with Microsoft .NET Framework 3.5. The 
rest of the code in this method walks through each of 
the controls and loads certain properties of each control 
into a property class and then places that class into a 
generic list. This gives you a list of all the controls (along 
with the type of the control) that you can walk through 
with .NET to set up the data binding. 

After the call to GetControlDefinitions, the Load- 
ControlsToGrid method has a generic list of all the 
XAML controls. The remaining code in this method 
walks through the items in the generic list and adds 
the valid ones to the DataGridView. The method 
determines which controls can be bound by calling the 
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Figure 2 


Two projects in the tool 


IsValidBindingType function, which returns True if the 
control’s type matches one in the Select statement of 
the IsValidBindingType function. 

Now you have enough code to load all of the con- 
trols that can be bound that are in a XAML form into 
a DataGridView. You need to feed in the source of the 
data you want to display to allow the user to match up 
the data fields with the controls on the form. The data 
field to match to a control is selected by the user 
from the Data Source Type ComboBox. The list of 
items a user can choose in this combo box is set as 
either a DataTable (by allowing the user to select a 
SQL Server table) or Class (property class). 

If users choose Class from the Data Source 
Type ComboBox, they will be prompted to select 
a property class. Then the tool will read the prop- 
erty class by calling the ProcessClassFileAnd- 
LoadProperties method in the DataBindingClass. 
This method simply reads the class file and finds 
all the property definitions and places them in a 


1 i EFFICIENT DATA BINDING 


AUTOMATING DEVELOPMENT CAN 
LEAD TO PROJECT SUCCESS 


I’ve found that the more automation | can build into the development process, 
the smoother the process works and the higher the likelihood is that it will 

be successful. | spend my most of my time working on large projects with 
complex applications. Often I'm called in to help a team get a project back on 
track. Project failure isn't as rare as you might think, see www.it-cortex.com/ 
Stat Failure Rate.htm. One of the most valuable lessons I’ve learned from put- 
ting projects back on track is the critical importance of automating the build/ 
test/deploy/install cycle. A fast and reliable build/test/deploy/install cycle allows 
you to move the application along at a fast pace so testers can see development 
changes quickly and the systems administrators who need to deploy and install 
aren't kept waiting. It's critical to surface errors quickly. In fact, once the build/ 
test pace spins up, it becomes clear where the design isn't right or where devel- 


opment isn't meeting design goals. | find that it often 


takes from 30 minutes 


to several hours to get an application up and running after the build. Even then 


it's hit or miss whether the application runs smoothly 


due to the complexity of 


the steps needed to set it up. Automating these processes makes the deploy- 
ment of the application a point-and-click repeatable process that puts the new 


build out for testers very quickly. 
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If users select DataTable from the Data Source 
Type ComboBox, the tool displays other fields that 
point to the SQL Server table to bind to, as Figure 3 
shows. Now you can enter a connection string and click 
the check mark to retrieve the list of tables from the 
database. Then you have just select a table from Tables 
ComboBox, and you can wire up the form. 

The tool uses the MetaDataSQLServerSimple class 
to extract metadata from SQL Server, so it's easy to pull 


I hope you'll enjoy working with this automation tool, 
but it's just a starting point. For instance, you could add 
stored procedures to the tables so a developer could select 
a stored procedures output. Currently, the tool generates 
only Visual Basic .NET code, but you could modify it 
to generate C# code. Or, instead of wiring up WPF, try 
wiring up Silverlight 2.0. You could add many other 
features. If you want to change the tool, have at it. EI 


Data binding tool with 
a database selected 


back information like table names and table fields. InstantDoc ID 100362 
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LINQPad 


[ 9 m a big fan of Microsoft Visual Studio, but 
sometimes it's easier to fire up a light- 
weight tool, try some code, and use those results in 
my main development project. When it comes to any 
programming that includes LINQ, my tool of choice is 
LINQPad (www.linqpad.net). LINQPad is free, small 
(2MB), downloadable, and requires no installation. 


More Than Just LINQ 

The LINQPad utility was written by Joseph Albahari, 
who along with his brother Ben Albahari co-authored 
CH 3.0 in a Nutshell. Albahari includes 200 examples 
from this book in LINQPad. At the heart of LINQPad 
is a standard text editor that lets you quickly type single 
expressions or many statements, execute them, and get 
nicely formatted output to look over. 


Queries 

LINQPad handles any valid Microsoft .NET Frame- 
work 3.5 C# statements against collections of any kind 
(such as in-memory collections, XDocument-based 
XML, or SQL Server-based objects). The LINQPad 
samples don’t require that the code be based on LINQ; 
code can be based on any C# code snippet that you 
want to try out. If you want to extend the tool, you 
can add references to your own assemblies and build up 
your LINQ queries against those objects. I’ve done this 
often to iteratively work on queries against the busi- 
ness objects in my Windows Presentation Foundation 
(WPF) VS.NET-based solution. It’s so much quicker 
to use LINQPad to write the query, test it, get it right, 
and then paste it into my real code. 


Output 

LINQPad displays formatted output that shows the 
results of your executed code. I like the hierarchical, 
treeview output for collections that you can expand 
and collapse as desired. Each node shows the object 
type and the count of items in a collection. In addition, 
you can see any embedded objects in the object graph 
and you can expand or collapse these objects, which 
gives you a visual feel for your data model. 

There are three handy buttons you can use to see dif- 
ferent types of output. The Results button is selected by 
default; it shows the graphical output of the query. The 
lambda operator shows the lambda equivalent of any of 
your LINQ queries that are IQueryable. The SQL button 
shows you any SQL statements that are executed based 
on the LINQ to SQL statements you executed. You can 
learn some complicated SQL by inspecting this output, 
as the following example shows: 
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SELECT [t@].[CustomerID], [t1].[ProductID] 
FROM [Orders] AS [t£], [Order Details] AS [t1] 
WHERE [t@].[OrderID] = [t1].[OrderID] 


Wish List 

One of the shortcomings of LINQPad is the lack of 
auto completion in the editor. Many developers have 
come to rely on the IntelliSense offered in Visual Studio. 
Most of the time, I use LINQPad as a way to get the 
syntax right for a complicated statement or explore 
new possibilities with queries, and it's very helpful to 
have the available members on a particular object right 
there in an auto completion list. The LINQPad com- 
munity has been asking for IntelliSense on the support 
forums, and Albahari is listening—IntelliSense is in the 
works. A beta switch in the current version shows you 
that he's working on it. Run LINQPad with a param- 
eter of -beta and you'll have IntelliSense: 


LINQPad.exe -beta 


As indicated on the support forums (forums.oreilly 


.com/category/22/C-3-0-in-a-Nutshell), only C# 2.0 


features are currently supported. 

One more feature I hope Albahari adds in a future 
release is the ability to customize the keyboard short- 
cuts available throughout the application. Shortcuts 
such as Ctrl+K, to comment a block of code, and U, to 
uncomment a block of code, while close to the Visual 
Studio equivalents, throw me off just enough to misuse 
them quite often. I'm such a keyboard-centric user that 
I want to map the shortcuts to those I use in Visual 
Studio 2008. I'd like to see keyboard commands to 
expand or collapse nodes in the output window added 
as well. SQL 

InstantDoc ID 100317. 


LINQPAD 


Dan Hanan 
dhanan @ interknowlogy.com) is a lead 


software engineer at InterKnowlogy, working 
with Microsoft .NET Framework, web services, 
and rich clients using Windows Presentation 
Foundation. He's a graduate of Stanford 


University. 


Pros: Free; no installation required; supports LINQ (e.g., objects, XML, 


SQL) and any other Microsoft .NET code; references external assemblies; has 
formatted hierarchical output; comes with lots of good documentation 


Cons: No IntelliSense in the current editor (it's planned for a future version); 
keyboard shortcuts are different from Visual Studio and not customizable 


Rating: X We ie We yy 


Price: Free download 


Recommendation: Definitely download and try if you're working with LINQ 


or just want a fast code snippet editor. 


Contact: Joseph Albahari e www.lingpad.net e Phone 


(Australia) 04198 23232 (International) +61 4198 23232 
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55% of SQL Server Magazine subscribers visit sqlmag.com on a 
monthly basis. 


4076 pass along their issues of SQL Server Magazine to at least 
one additional person. 


It delivers technical, tactical, and industry information that 
empowers DBAs, developers, and Bl architects to do their 
jobs better. 


6276 of readers spend 1-3 hours reading a typical issue. 


SQL Server Magazine features 10 SQL Server MVPs as either 
contributing editors or regular authors. 


$596 million reasons (average subscriber revenue). 


70% have taken further action as a result of seeing 
an ad or article. 


$3.34 million reasons (average software and 
hardware budgets per reader). 


9076 of our readers are involved in the purchasing decisions. 


SQL Server Magazine is the ONLY paid publication focused on 
SQL Server. 


Readex Reader Survey, September 2007 


Contact info: 

Birdie Ghiglione 
619-442-4064 
Birdie.ghiglione@sqlmag.com 


RowGen 2.0 


[ today's world of Health Insurance Portability 
n and Accountability Act (HIPAA) and privacy- 
related regulations, the ability of Innovative Routines 
International's RowGen 2.0 to replace sensitive client 
information copied from your production environment 
for use in your test environment with realistic data that 
doesn't infringe on your clients’ privacy is very useful. 
RowGen can take a file definition of your data struc- 
tures and use the structures as a template to create refer- 
entially valid but random test data for your applications. 
This product is compatible with data definitions from 
multiple data stores, including SQL Server, Oracle, DB2, 
and packaged applications such as PeopleSoft. You 
can load the file containing your DDL into RowGen 
and it will automatically generate data based on your 
definition. The idea sounds promising; however, let me 
state up front that RowGen isn’t a simple, easy-to-use 
product. 

You can’t just install and run RowGen because it 
requires a significant time investment to learn its pro- 
prietary data definition syntax, so you'll want to have 
someone dedicated to working with the tool. During 
the installation process, three executables were installed 
on my test machine. The first executable, Setup, is for 
defining your license; the second is RowGen itself, a 
command-line executable that doesn't have a menu 
item in Windows; and the third is the RapidACE- 
RowGen GUI executable. (RapidACE is the front end 
for the RowGen engine.) Once you've installed the 
package, run the Setup program to generate the text 
file that defines your system's characteristics so that 
you can be issued a license from CoSort, which you 
need to run the tool. 

The product's software also adds a set of configura- 
tion settings to the registry under HKEY LOCAL 
MACHINE\SOFTWARE\Innovative Routines Int'l, 
Inc.\RowGen 2.1\Global Configuration. These settings 
include the number of threads that are limited by 
your license, and if you get a runtime error related to 
exceeding the number of available sort threads, you'll 
need to edit these settings. Note that although the 
settings can be overridden in a local 
resource file, I recommend updating 
the settings here. 

There are three main panes in 
RapidACE, as shown in Figure 1. 
On the left side of the RapidACE window is the 
Workspace pane in which you can identify the files that 
contain your data generation rules. The top right pane 
is a text edit window in which you can edit the contents 
of your data generation files (i.e., either an imported 
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DDL file or your generated script—.rcl—files). The 
bottom right window is the Console pane in which 
you can execute specific data generation commands. 
One of the first things I realized as I worked with the 
tool is that the reason it's database agnostic is because 
it works with files only on the file system, thus you 
need to export your T-SQL. Note that when exporting 
T-SQL you should export only the table definition 
because any additional information is likely to confuse 
RowGen and keep it from recognizing your tables. 
Considering the fact that Microsoft's Visual Studio 
Team System 2008 Team Suite is going to include a sim- 
ilar tool, the amount of time needed to master RowGen 


William Sheldon 
bsheldon @ interknowlogy.com) is a con- 


tributing editor for SQL Server Magazine, a 
principal engineer with InterKnowlogy, and 
a Microsoft MVP for Visual Basic .NET. He's 


coauthor of Professional Visual Basic .NET 


See the full review and web 
listing at InstantDoc ID 100303. 5 OS COLUMN PXLOTEE EYALUATIONS.REMEDIATION FLAS IJ "This field represents 


and RapidACE, as 
well as set up all of 
the required DDL 
and script files, might 
make RowGen not 
worth its four-figure 
price. However, if you 
have a heterogeneous 
database environment 
and a dedicated test 
team that's familiar 
with test automation 
and writing scripts, 
this tool's command- 
line capabilities might 
be of interest to you. 

SQL 


InstantDoc ID 100303 


with .NET 3.0 (Wrox) and an instructor at 
University of California, San Diego. 


ROWGEN 2.0 


Pros: Creates random test data; scriptable 


Cons: Custom scripting syntax; GUI is limited and 
its behavior is unintuitive; it doesn’t place the data 
in your database; there were setup problems 


Rating: WS yy yy 


Price: Contact a sales representative for pricing 
information. 


Recommendation: If you have a large organiza- 
tion, a heterogeneous environment, a full-time test 
team with scripting experience, and the need to 
perform complex data-driven testing using simu- 
lated real-world data, this tool will meet your needs. 


Contact: Innovative Routines International e 
321-777-8889 G www.cosort.com 
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Cons: Offers weak diagnostic capabilities and 
no drill-down; supports extending out-of-box 
capabilities using custom reports, Checks, or 
Tasks, but doing so takes quite a bit of effort 


Rating: KAAK 


Price: Starts at $195 for Standard Edition 
(monitoring for five servers) and $1,995 for a 
Professional Edition 10-server license; $14,995 
for Enterprise Edition (unlimited licenses). 


Recommendation: dbWatch 8.1 is an effective, 


DBWATCH 8.1 


Pros: Provides basic database monitoring 
for SQL Server, Oracle, and MySQL; 
inexpensive; provides integrated cross-platform 
reporting and alerting; easy to use 


dbWatch 8.1 


D you need to monitor databases without 
Qs your budget? If so, dbWatch 


Software's dbWatch 8.1 provides such capabilities. It's 
a multiplatform, economical solution for keeping tabs 
on databases. It takes a minimalist approach to data- 
base monitoring, alerting, and reporting. dbWatch 
8.1 supports SQL Server 2005/2000, Oracle 8 and 
later, and MySQL. The vendor has announced it will 
support SQL Server 2008 in late 2008 as well as Post 
greSQL and Sybase in future point releases. I reviewed 
dbWatch Professional Edition; other editions include 
dbWatch Standard Edition, and Enterprise Edition. 


dbWatch Features 

dbWatch has a distributed architecture, which includes 

these components: 

* dbWatch Engine: a schema that's installed on each 
monitored server and provides the framework for 
dbWatch Tasks and Checks. 

* dbWatch Tasks/Checks: SQL-based plug-ins to 
the dbWatch Engine. Both the Tasks and Checks 
plug-ins record trend-analysis data to a database 
on the monitored server. The Checks plug-ins also 
alert you in real time to potential issues, such as low 
memory or job failure. 

* dbWatch Server: a Java application that runs on any 
Java-supporting platform and initiates the execution 
of Tasks and Checks following a schedule you set. 
dbWatch Server can run as a service or daemon but 

must be active to run the 

Tasks and Checks. 

* dbWatch Extensions: 

a component that lets 
dbWatch Server com- 
municate with other 
Systems. 

* dbWatch Monitor: 
the UI that you use 
to install dbWatch 
Engine, configure 
Tasks and Checks on 
monitored database 
servers, review server 
status in real time, 
and use the reporting 
system. 


no-frills database monitoring system that | recom- 
mend for budget-conscious users who need basic 


monitoring capabilities. 


Using dbWatch 


An installation guide in 


dbWatch architecture and procedures to install 
dbWatch Server and dbWatch Engine. dbWatch Server 
installed quickly on my Windows Server 2003 test 
system. From dbWatch Monitor, the Register Data- 
base wizard installs dbWatch Engine on monitored 
database servers. The wizard also installs a default set 
of 10 Tasks and 6 Checks. 

dbWatch Monitor has a fairly standard layout, let- 
ting you organize supervised systems by user-defined 
groups. You can also see all the standard Tasks and 
Checks available for SQL Server. What you see in 
Monitor is all you get. The default Checks survey some 
basic parameters, such as disk space, memory usage, 
job execution, and blocked transactions. The icons dis- 
played in Monitor are color coded according to the error 
severity at any level in the display hierarchy, providing 
a quick, high-level overview of your database servers’ 
health. Green, for example indicates that all is OK, while 
yellow indicates a task or check has issued a warning. 

dbWatch Monitor provides an SQL Worksheet, 
which lets you execute SQL statements against one 
or more monitored servers simultaneously by simply 
checking a box for the server or group name. dbWatch 
will merge the results into a table, which you can save 
to a text file. 

English-language documentation for dbWatch 
is still a work in progress. In addition to the instal- 
lation guide, dbWatch includes an HTML-based 
Users Guide, which I found somewhat helpful but 
incomplete—lacking, for example, an alphabetic index 
and text-search capability. 

dbWatch Monitor has a wizard-driven integrated 
reporting feature that’s easy to use and lets you produce on- 
demand or scheduled reports. dbWatch includes five 
standard reports for SQL Server users, including avail- 
ability statistics and a SQL Server health report. dbWatch 
generates reports in either PDF or HTML format. 


Helpful and Low-Cost 

dbWatch is an effective, easy-to-use multiplatform 
database-monitoring tool. Although it does not include 
the diagnostic capabilities of database-monitoring 
products from Idera or Quest Software, at less than 
$200 per monitored server, it provides serviceable 
monitoring at only a fraction of the cost of those tools. 
Scheduled reporting helps you spot trends in database 
disk and memory utilization, and email or SMS alerting 
quickly apprises you of issues. If you're looking for a 
basic, budget-friendly database-monitoring solution, 


Contact: dbWatch Software e www.dbwatchsoft 


ware.us e (47) (22) 33-14-20; 800-270-9892 dbWatch should meet your needs. gu 
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Bytes from the Blog 
www.sqlmag.com/go/industrybytes 


The London Stock Exchange Crash: 
TradElect's Rise and Fall 


o September 8, the Monday after the US 
n government announced that it was stepping 
in to assume control of Freddie Mac and Fannie Mae, 
traders and investors on the London Stock Exchange 
(LSE) were about to reap the rewards of the surging 
markets reacting to the news. However, things soon 
went awry as the LSE suffered a technology glitch that 
halted trading for more than seven hours. 

Information about what caused the crash 
has been slow to emerge, although the LSE told 
ComputerworldUK that the problem was related to net- 
work software and not the LSE's TradElect platform, 
which was developed with products and consulting 
advice from a variety of technology and consulting 
firms, including Microsoft, HP, Accenture, Cisco 
Systems, and Avanade. 


Dissecting TradElect 
So what exactly is TradElect? According to the LSE, 
TradElect is a platform developed using Microsoft 
.NET Framework that's significantly faster, more 
reliable, and more functional than the LSE's previous 
trading platform. Several companies publicized their 
roles in building the TradElect system. For instance, 
HP issued a press release on July 1, 2006, with the 
headline ^London Stock Exchange Becomes World's 
Fastest with HP and Microsoft Technology." Avanade, 
a joint venture between Microsoft and Accenture, was 
also involved with developing trading solutions for the 
LSE, including the TradElect platform, and published 
a case study about its LSE work. In the case study, the 
firm described how it used SQL Server 
2000 and Windows 2000 Advanced 
Server and Datacenter Server for the 
LSE’s RSP Gateway. 

But Microsoft promoted its involve- 


LSE deal in a variety of advertisements, including an 
ad representing a fake newspaper called “The Highly 
Reliable Times” that featured the headline “London 
Stock Exchange Chooses Windows over Linux for 
Reliability.” Perhaps Microsoft’s biggest gaffe in the 
TradElect project was producing a promotional video 
about the LSE’s decision to move its trading platform 
to Windows Server. It’s a short video clip filled with 
quotes and assurances about the reliability and perfor- 
mance of Windows Server. 


The Aftermath 

Although actual details of the situation have been slow 
to emerge, the fact that Microsoft (and by extension, 
its partner Accenture and joint venture Avanade) 
invested so heavily in promoting involvement with 
the LSE puts Microsoft in the unenviable position 
of choosing between two bad options: Either accept 
that a high-profile Microsoft IT solution experienced 
a major technical problem, or let the blame fall on the 
individuals managing the platform, thereby exposing a 
valuable customer to the ire of angry stock traders. The 
LSE trading-system crash has proved that although 
customer case studies and white papers featuring high- 
profile clients can be valuable sales tools, they can also 
bring a company negative publicity. 

For links to the relevant case studies and news 
articles discussed in this article, visit the original blog 
post at www.sqlmag.com, InstantDoc ID 100260. ER 
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ment with the LSE more than any vorcwi-msuR! — windows Server's — | special edition 
other tech vendor, with marketing col- LONDON STOCK EXCHANGE "We looked at a number of different plat- 
lateral that included white papers extol- CHOOSES WINDOWS OVER forms for our Technology Roadmap...and 
: 3 Windows Server was the clear choice.” 
ling the benefits of moving the LSE P 


trading platform from a Linux back 
end to a solution based on Windows 
Server 2003. Microsoft also used the 
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ments to products @ 
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—Jeff James, 


senior editor 
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release of Tableau 4.0, its analytics and visualiza- 
tion product. Tableau 4.0 includes updates to Tableau Server and Tableau Desktop, two products that increase 
data visualization and information sharing. Tableau 4.0 offers the ability to create an interactive analytical map 
with one click and share live Tableau views with web applications and intranets. This product also includes visual 
cues to show feature capabilities, as well as the functionality to sort your work by visual representations. For more 
information, call 206-633-3400 or visit www.tableausoftware.com. 


APPLICATION DEVELOPMENT 

RAD Tool Now Includes Support for SQL Server 2008 

Sybase has announced the latest version of PowerBuilder, its 4GL rapid application development (RAD) tool. 
PowerBuilder 11.5 lets you develop applications and deploy them to .NET and provides native driver support for 
SQL Server 2008. The latest version also includes new functionality for creating visually appealing applications, 
enhanced deployment options via an Application Server plug-in, and support for the Federal Desktop Core 
Configuration (FDCC) security standard and Code Access Security (CAS), which lets . NET applications run in 
partially trusted zones. PowerBuilder costs $2,995 for new licenses and $1,495 for updates to PowerBuilder 11.0. 

For more information, call 925-236-5000 or visit www.sybase.com. 


APPLICATION DEVELOPMENT 

Delphi 2009 and C++Builder 2009 

Support Database Development 

Embarcadero Technologies has announced two rapid application develop- 
ment tools (RAD) for Windows— Delphi 2009 and C++Builder 2009—that 
are designed for application developers and database developers who need 
an IDE with a heterogeneous database framework. The professional edi- 
tions for both products include local database connectivity to InterBase, 
Blackfish SQL, and MySQL. The enterprise editions also offer connectivity 
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to SQL Server, Oracle, DB2, Informix, and Sybase ASE, plus Data Snap 
multi-tier database application development. Upgrade pricing begins at $374 per license for Professional editions. 
For more information, call 415-834-3131 or visit www.codegear.com/products. 


PERFORMANCE MONITORING 

New Features Added in Performance Center 2.5 

Embarcadero Technologies has released a new version of its performance-monitoring product, Performance 
Center, with features that include historical and real-time database performance analysis, a health index perfor- 
mance indicator that shows the performance of each monitored database, and tools for detailed reporting to 
various audiences, including CIOs and IT managers. With Performance Center 2.5, you can monitor SQL Server, 
DB2 for LUW, Oracle, and Sybase ASE from a single interface. Introductory pricing begins at $2,650 per user. 
For more information, call 415-834-3131 or visit www.embarcadero.com. 


SECURITY 

Stop SQL Injection Attacks with UrlScan 3.0 

UrlScan is a free application that works with Microsoft's IIS web server to combat SQL injection attacks. 
Microsoft has released UrlScan 3.0, which offers a number of new features, including support for query 
string scanning. In addition, UrlScan 3.0 lets users specify safe URLs that can bypass scanning and also 
includes a number of new fields to improve analysis of log files. For more information, visit www.microsoft 


.com/WindowsServer2003/IIS/Default.mspx and click the IIS Downloads link. E 
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Microsoft? SQL Server" 2005 e Business Intelligence e More 


Call 800-578-2062 or visit www.ksourceit.com for a free trial! 


NEED IT. LEARN IT. KNOW IT. 


KSOURCE IT: 
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format customer data 
at the point of entry 
or in batch. Easily 
integrate with .Net, 
MS SQL or Java. 
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Rancho Santa Margarita, CA 
92688-2156 

Carrier Route: C056 
CountyName: Orange 


TimeZone: Pacific 


Delivery Indicator: Business 


Suite Status: Validated 
Latitude: 33.6480 
Longitude: -117.6000 


(and more) 


Get a FREE demo at: 
MelissaData.com/sql 
1-800-MELISSA 


MELISSAIDAT A 


Your Partner in Data Quality 


A x 
“Chnical learning 


November 2008 


Michael Otey 


motey@ sqlmag.com) is technical director 
for Windows IT Pro and SQL Server Magazine 


and author of Microsoft SQL Server 2008 New 
Features (Osborne/McGraw-Hill). 


am Microsoft develops each new release of 
Ð SQL Server, it seems compelled to add new 
editions and SQL Server 2008 offers several. Here's 
a quick look at SQL Server 2008 editions. 


Top dog of the SQL Server product line, Enterprise 
Edition possesses the entire SQL Server 2008 feature 
set. It supports the OS maximum of 64 CPUs, 2TB 
of RAM, and database size limited only by available 
storage. Notable features exclusive to Enterprise Edi- 
tion include support for data compression, transparent 
database encryption, and unlimited virtual machine 
licensing. It’s priced at $24,999 per processor or 
$13,499 per server including 10 CALs. 


Although it’s got the same features as Enterprise Edition, 
Developer Edition is licensed only for developmental use. 
It isn’t used as a production database server. 


New to the SQL Server lineup, Web Edition is designed 
to be run by web-hosting providers. It supports up to 
4 CPUs, 2TB of RAM, and unlimited database size. 
BI-wise, it supports SQL Server Reporting Services 
(SSRS). It's priced at $15 per processor per month. 


Standard Edition provides c core database and business 
intelligence (BI) feature sets. It supports up to 4 CPUs 


and 2TB of RAM, with unlimited database size. Sup- 
port for BI subsystems includes SQL Server Analysis 
Services, SQL Server Integration Services, and SSRS. 
It’s priced at $5,999 per processor or $2,799 per server 
with 10 CALs. 


Workgroup Edition is for departments and branch 
offices. It supports a maximum of 2 CPUs, 4GB of 
RAM, and unlimited database size. Like Web Edition, 
its only BI support is for SSRS. It does include the 
SQL Agent. Workgroup Edition is priced at $3,899 or 
$739 per server including five CALs. 


Designed for small workgroups, Express Edition is 
often used as the built-in database for Microsoft Share- 
Point and System Center products. A free download, 
it supports a maximum of 1 CPU, 1GB of RAM, 
and database sized to 4GB. Express Edition with 
Advanced Services adds SQL Server Management 
Studio Express and support for SSRS. 


Built on a different code base than the other editions, 
SQL Server Compact Edition comes in Microsoft 
Visual Studio 2008. It’s a small-footprint, in-process 
database with limited relational database capabilities. 
SQL 

InstantDoc ID 99382 


EE hese days, change seems to be everywhere but 

| in my pocket. In this years US presidential 
campaign, | heard a lot about change. And now, | 
can’t escape the ever-increasing talk about upgrading 
to SQL Server 2008. So I've drawn a line in the sand: 
Just because something's new, you don't have to 
leap to adopt it. You need to carefully compare SQL 
Server 2008 to the version that you're running now, 
whether it's SQL Server 2005 or SQL Server 2000. 
Even if you decide not to upgrade, Sglmag.com still 
has the information you need to keep you going with 
whatever SQL. Server version you have. And unlike 
some political slogans, these are solutions you can 
believe in. | promise. 


SQL Server 2005: 

"SQL Server 2005 Resources, " 

InstantDoc ID 97839 

"Leverage Reporting Services 2005 Independent 
of Your Database," InstantDoc ID 48629 


"SQL Server Consolidation," InstantDoc ID 95461 


"SQLQueryStress," InstantDoc ID 97906 
"Best Practices Analyzer," InstantDoc ID 95383 


SQL Server 2000: 

"Bare-Metal Tuning," InstantDoc ID 46492 
"Moving Cubes from Analysis Services 2000 
to 2005,” InstantDoc ID 95161 
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Humphries 


(christan.humphries@ penton.com) is a 


Windows IT Pro associate and regular 
contributor to Windows IT Pro, SQL 
Server Magazine, and associated Web 
sites, specializing in the Windows IT Pro 
network of tools and resources. 
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SQLSENTRY | 
PerformanceAoVvsor " 


The New Standard in Monitoring and Performance 


* Performance Dashboard with relevant SQL Server 
and Windows metrics 


* Real-time and historical performance analysis 

* Disk activity, latency, and capacity monitoring 

* Top SQL analysis highlights heaviest queries 

* Graphical blocking and deadlock analysis 

* Calendar views of Top SQL, blocks and deadlocks 

* One-click and automated tracing with Quick Trace™ 


Free Trial Download: sqlsentry.net/sgl-performance SENTRY 


OVER 10,000 


“Our auditors visit quarterly and have been very pleased with the level of tracking and 


reporting that we have — thanks to SOL compliance manager.” 


COMPLIANCE 


“SQL compliance manager is a perfect solution for auditing high-transaction servers. 


It allows us to monitor any changes and attempted changes by users, date and system.” 


AUDITS PASSED. 


* Unlike other products, SOL compliance manager was ready-to-use right out of the box! 


But it is also very easy to customize in the future if we wish.” 
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Peace of mind, brought to you by Idera! v v + MANAGING YOUR WINDOWS WORLD ™ 

Start today with your 14-day free trial of Idera's award-winning 

SOL compliance manager. Track who's doing what on your SOL SERVER 

SQL Servers and be ready when your next database audit 

SHAREPOINT 
comes around. spews 
"A MID POWERSHELL 
v 
G Capture, audit & alert on SOL Server activity: DYNAMICS 


Download SOL compliance manager free 14-day trial today! 


www.idera.com/SQLcompliance 


